Re: [Haskell-cafe] Takusen sqlite3 insert is very slow
Hi Jason, Your recommendations worked for me. When I enclosed updating into single transaction, the code executed in less than 0.5 seconds, which is as fast as HDBC version. I didn't go deeper, hoping, that everything will be OK from now. Thank you, Vasyl 2010/3/20 Jason Dagit da...@codersbase.com: On Sat, Mar 20, 2010 at 3:32 AM, Vasyl Pasternak vasyl.paster...@gmail.com wrote: Hi Cafe, I have another problem, please look at code: storeInDb = withSession (connect test.db) (do execDDL (sql create table x (y int)) forM_ ([1..1] :: [Int]) (\x - do execDML (cmdbind (insert into x (y) values (?);) [bindP x]) return ())) This code runs 16 seconds which is very slow for this simple task. RTS output is below. After profiling this program I found that 85% of its time it spends in 'Database.Sqlite.SqliteFunctions.stmtFetch'. Currently I don't know how to make it faster, maybe anyone had this problem later? HDBC inserts very fast, so this is not sqlite error. Can you show the HDBC version? Maybe they make different assumptions about transactions or fetching the number of affected rows? If I'm reading the source of takusen correctly it's using a different transaction for each insert and stmtFetch is getting called to return the number of rows inserted. Which should be 1 every time and for your purposes, ignorable. You should be able to change to execDDL, but I seriously doubt that will have any impact on performance. It looks like the only difference between execDDL and execDML is that execDDL has return () at the end of it. You might try running your inserts inside withTransaction. The default behavior of sqlite is to use a separate transaction for each statement. Perhaps this is adding overhead that shows up during stmtFetch. How long does your HDBC version take? Is it a factor of 10? Factor of 2? Jason ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Takusen and iteratee problem
Hi, seems, that I found how to solve it. I replased `mtl` dependency with `transformers` and `monads-fd` and everything works fine. BTW. Iteratee and Takusen didn't want to work because of different instances of MonadIO (Iteratee needs MonadIO from transformers package, but Takusen implements MonadIO from mtl) Regards, Vasyl 2010/3/20 Valery V. Vorotyntsev valery...@gmail.com: Today I stuck with the following problem: I want to read a file with iteratee package, and but it to database through Takusen package, but it doesn't work. The `Takusen` was built with `mtl` package, and `iteratee` - with `transformers`, so they are conflicting when used simultaneously. And what error message is displayed? -- vvv ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Takusen sqlite3 insert is very slow
Hi Cafe, I have another problem, please look at code: storeInDb = withSession (connect test.db) (do execDDL (sql create table x (y int)) forM_ ([1..1] :: [Int]) (\x - do execDML (cmdbind (insert into x (y) values (?);) [bindP x]) return ())) This code runs 16 seconds which is very slow for this simple task. RTS output is below. After profiling this program I found that 85% of its time it spends in 'Database.Sqlite.SqliteFunctions.stmtFetch'. Currently I don't know how to make it faster, maybe anyone had this problem later? HDBC inserts very fast, so this is not sqlite error. RTS output: 156,195,264 bytes allocated in the heap 591,280 bytes copied during GC 3,952 bytes maximum residency (1 sample(s)) 10,968 bytes maximum slop 1 MB total memory in use (0 MB lost due to fragmentation) Generation 0: 295 collections, 0 parallel, 0.00s, 0.01s elapsed Generation 1: 1 collections, 0 parallel, 0.00s, 0.02s elapsed INIT time0.00s ( 0.03s elapsed) MUT time0.87s ( 16.09s elapsed) GCtime0.00s ( 0.02s elapsed) RPtime0.00s ( 0.00s elapsed) PROF time0.00s ( 0.00s elapsed) EXIT time0.00s ( 0.00s elapsed) Total time0.88s ( 16.14s elapsed) %GC time 0.5% (0.1% elapsed) Alloc rate179,111,917 bytes per MUT second Productivity 99.5% of total user, 5.4% of total elapsed Regards, Vasyl ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Takusen and iteratee problem
Hello Cafe. Today I stuck with the following problem: I want to read a file with iteratee package, and but it to database through Takusen package, but it doesn't work. The `Takusen` was built with `mtl` package, and `iteratee` - with `transformers`, so they are conflicting when used simultaneously. To be concrete, I want to run the code, that looks like: withSession (connect dbname.db) (do execDDL (sql create table x (value string)) fillFromFile file.dat ) fillFromFile fname = fileDriverFd iter fname iter = iterator that uses execDML (sql Insert into ) I have no ideas how to solve the problem, so appreciate any recommendations, that will make it possible. Thanks, Vasyl ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Iteratee performance
Hi Cafe, Yesterday I played with iteratee package, and wanted to check its performance. I tried to count lines in a file, as Oleg in his famous lazy_vs_correct[1] article. The results somewhat disappointed me. The statistics and code follows, but shortly: lazy bytestring is the fastest, iteratee with bytestrings 10 times slower than lazy bytestring. When comparing lazy string and iteratee with [Char], than their results were close, but lazy string reading uses less memory and was a bit faster (20%). I performed test on 250Mb file with 5 millions lines. Now I am figuring out, is these tests are correct and this is ordinary behavior, so iteratee not so fast as I thought, or there is some mistake in my code. [1] http://okmij.org/ftp/Haskell/Iteratee/Lazy-vs-correct.txt - TIMING RESULTS -- $ time wc -l 500.txt 500 500.txt real0m0.314s user0m0.184s sys 0m0.124s $ time ./bytestring_test 500.txt 500 real0m0.817s user0m0.616s sys 0m0.200s $ time ./bytestring_iteratee_test 500.txt real0m7.801s user0m7.552s sys 0m0.252s $ time ./string_test 500.txt 500 real0m47.427s user0m46.675s sys 0m0.648s $ time ./string_iteratee_test 500.txt 500 real0m59.225s user0m57.680s sys 0m0.840s -- RTS INFO ./bytestring_test 500.txt +RTS -sbs.out 807,225,096 bytes allocated in the heap 122,240 bytes copied during GC 59,496 bytes maximum residency (1 sample(s)) 22,424 bytes maximum slop 1 MB total memory in use (0 MB lost due to fragmentation) Generation 0: 1540 collections, 0 parallel, 0.03s, 0.02s elapsed Generation 1: 1 collections, 0 parallel, 0.00s, 0.00s elapsed INIT time0.00s ( 0.00s elapsed) MUT time0.59s ( 0.79s elapsed) GCtime0.03s ( 0.02s elapsed) EXIT time0.00s ( 0.00s elapsed) Total time0.62s ( 0.82s elapsed) %GC time 4.5% (2.8% elapsed) Alloc rate1,372,743,081 bytes per MUT second Productivity 95.5% of total user, 72.1% of total elapsed - ./bytestring_iteratee_test 500.txt +RTS -siter.out 11,024,100,312 bytes allocated in the heap 893,436,512 bytes copied during GC 95,456 bytes maximum residency (1 sample(s)) 23,216 bytes maximum slop 1 MB total memory in use (0 MB lost due to fragmentation) Generation 0: 21030 collections, 0 parallel, 2.51s, 2.45s elapsed Generation 1: 1 collections, 0 parallel, 0.00s, 0.00s elapsed INIT time0.00s ( 0.02s elapsed) MUT time6.37s ( 6.66s elapsed) GCtime2.52s ( 2.45s elapsed) EXIT time0.00s ( 0.00s elapsed) Total time8.88s ( 9.12s elapsed) %GC time 28.3% (26.9% elapsed) Alloc rate1,731,061,437 bytes per MUT second Productivity 71.7% of total user, 69.8% of total elapsed - ./string_test 500.txt +RTS -sstr.out 38,561,155,264 bytes allocated in the heap 9,862,623,816 bytes copied during GC 223,080 bytes maximum residency (5026 sample(s)) 47,264 bytes maximum slop 2 MB total memory in use (0 MB lost due to fragmentation) Generation 0: 68525 collections, 0 parallel, 22.50s, 22.51s elapsed Generation 1: 5026 collections, 0 parallel, 1.38s, 1.36s elapsed INIT time0.00s ( 0.00s elapsed) MUT time 22.80s ( 23.55s elapsed) GCtime 23.87s ( 23.87s elapsed) EXIT time0.00s ( 0.00s elapsed) Total time 46.67s ( 47.42s elapsed) %GC time 51.1% (50.3% elapsed) Alloc rate1,691,170,222 bytes per MUT second Productivity 48.9% of total user, 48.1% of total elapsed - ./string_iteratee_test 500.txt +RTS -sstriter.out 40,164,683,672 bytes allocated in the heap 7,108,638,256 bytes copied during GC 212,624 bytes maximum residency (821 sample(s)) 50,264 bytes maximum slop 2 MB total memory in use (0 MB lost due to fragmentation) Generation 0: 75791 collections, 0 parallel, 33.14s, 33.75s elapsed Generation 1: 821 collections, 0 parallel, 0.56s, 0.63s elapsed INIT time0.00s ( 0.00s elapsed) MUT time 23.99s ( 24.84s elapsed) GCtime 33.69s ( 34.38s elapsed) EXIT time0.00s ( 0.00s elapsed) Total time 57.68s ( 59.22s elapsed) %GC time 58.4% (58.1% elapsed) Alloc rate1,674,540,397 bytes per MUT second Productivity 41.6% of total user, 40.5% of total elapsed -- SOURCECODE - $ cat bytestring_test.hs import System.Environment import Control.Monad import qualified Data.ByteString.Lazy.Char8 as B count s = liftM (length . B.lines) (B.readFile s) main = do [f] - getArgs print = count f --- $ cat bytestring_iteratee_test.hs
Re: [Haskell-cafe] Iteratee performance
Gregory,
Thank you, your code helps, now my it runs in the speed of lazy
bytestring test but uses less memory with it.
I've only added to your code more strictness in the recursion, my
version is below.
BTW, I think it is more useful to let user set the chunk size for
reading, so I'd like to see this possibility in the iteratee package.
{-# LANGUAGE RankNTypes, ScopedTypeVariables #-}
import qualified Data.Iteratee.IO.Fd as I
import qualified Data.Iteratee as I
import qualified Data.Iteratee.WrappedByteString as I
import qualified Data.ByteString.Char8 as S
import System.Environment
import System.IO
count :: FilePath - IO Int
count s = I.fileDriverFd cnt s
cnt :: (Monad m) = I.IterateeG I.WrappedByteString Char m Int
cnt = go 0
where
go n = I.IterateeG $ \ch -
case ch of
(I.EOF Nothing)- return $ I.Done n ch
(I.EOF (Just e)) - return $ I.Cont cnt (Just e)
(I.Chunk (I.WrapBS s)) - do
let n' = n + S.count '\n' s
return $ n' `seq` I.Cont (go n') Nothing
main :: IO ()
main = do
[f] - getArgs
print = count f
2010/3/17 Gregory Collins g...@gregorycollins.net:
Vasyl Pasternak vasyl.paster...@gmail.com writes:
Hi Cafe,
Yesterday I played with iteratee package, and wanted to check its
performance. I tried to count lines in a file, as Oleg in his famous
lazy_vs_correct[1] article. The results somewhat disappointed me.
eris:benchmark greg$ time ./IterateeTest /usr/share/dict/words
234936
real 0m0.027s
user 0m0.010s
sys 0m0.015s
eris:benchmark greg$ time ./ByteStringTest /usr/share/dict/words
234936
real 0m0.024s
user 0m0.015s
sys 0m0.007s
Note also that the Bytestring I/O functions use a 32KB buffer and the
iteratee enumFd function uses a 4KB buffer; if the buffers were the
same the performance would be comparable. Here is my code:
{-# LANGUAGE RankNTypes, ScopedTypeVariables #-}
import qualified Data.Iteratee.IO.Fd as I
import qualified Data.Iteratee as I
import qualified Data.Iteratee.WrappedByteString as I
import qualified Data.ByteString.Char8 as S
import System.Environment
import System.IO
count :: FilePath - IO Int
count s = I.fileDriverFd cnt s
cnt :: (Monad m) = I.IterateeG I.WrappedByteString Char m Int
cnt = go 0
where
go n = I.IterateeG $ \ch -
case ch of
(I.EOF Nothing) - return $ I.Done n ch
(I.EOF (Just e)) - return $ I.Cont cnt (Just e)
(I.Chunk (I.WrapBS s)) - do
let n' = n + S.count '\n' s
return $ I.Cont (go n') Nothing
main :: IO ()
main = do
[f] - getArgs
print = count f
--
Gregory Collins g...@gregorycollins.net
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] How to define instance for type synonym with parameter.
Aren't ErrorT and ReaderT instances of Applicative already? (At least in 'transformers' package?) what difference between 'mtl' and 'transformers' packages ? 'mtl' goes with Haskell Platform as standard package, so what is the benefit of 'transformers' library ? Thanks, Vasyl ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] How to define instance for type synonym with parameter.
Hello, I am start with example. Suppose I have the following type synonym: type SomeMonad a = ErrorT String (ReaderT String IO) a this is monad, and I want to make it instance of Applicative, so, the obvious way is to write the following: instance Applicative SomeMonad where pure = return (*) = ap GHCi warns me, that I have to use -XTypeSynonymInstances option to allow this construction, but than I have following error: Type synonym `SomeMonad' should have 1 argument, but has been given 0 In the instance declaration for `Applicative SomeMonad' Neither `instance Applicative (SomeMonad a)` nor `instance Applicative SomeMonad a` help. But works the following: instance Applicative (ErrorT String (ReaderT String IO)) where pure = return (*) = ap Which is the same (from my point of view). Could anyone tell me what is going on, and how to declare SomeMonad as instance of Applicative ? Thanks, Vasyl ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] listing mountpoints and getting their properties in Haskell
Eugene, The only Linux function that can do this is `statvfs`. But binding to it weren't implemented in the current libraries. Year ago I implemented function that queries free size of the mounted filesystem. The file is attached. To get other parameters you should only extend `peek` function of the `Storable` class. Hope it helps. Vasyl 2010/2/27 Eugene Dzhurinsky b...@redwerk.com: Hello! I need to list all currently mounted filesystems and get some stats like total space, free space, mount point and physical device. Is there any library capable of obtaining such information from OS itself? Parsing output of 'df' is locale-dependent and error-prone (because of locale settings, output settings etc). Thank you in advance. -- Eugene N Dzhurinsky ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe statvfs.hsc Description: Binary data ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Generate random UArray in constant memory space.
Hi all, To summarize everything in this thread I've tested mwc-random, System.Random and mersenne random numbers (mersenne-random-pure64). Here the score table: [THIRD PLACE] Generic Random Number Generator. Is the slowest and allocates too much memory in the heap. The total memory usage is constant and very low. [SECOND PLACE] MWC-RANDOM. The fastest random number generator ever. But it uses O(n) memory for generate random numbers. Thought it isn't possible to calculate a really large set of random numbers (my PC stuck with calculating 500 millions random numbers with memory usage above 3,5Gb). The memory usage for me is more important than time, because I can easily wait additional 5-10-15 mins, but I cant so easily put additional memory to my PC. Thus this is only second place. [FIRST PLACE] Mercenne Random number generator. Approx 10 times faster than generic and two times slower than mwc. But it works in constant memory space, so theoretically it could generate infinite list of numbers. It also uses 6 time less total allocations, than generic RNG. NOTE: These tests didn't test the quality of the random sequences, only speed/memory. Thanks to everyone, who helped me with this code, it seems, that now I understand optimizations much better, than a day ago. Best regards, Vasyl 2010/2/10 Felipe Lessa felipe.le...@gmail.com: On Tue, Feb 09, 2010 at 04:27:57PM -0800, Bryan O'Sullivan wrote: It creates and returns a vector, so if you ask it to give you a billion items, it's going to require north of 8 gigabytes of memory. This should not come as a surprise, I'd hope :-) Assuming that's not what you actually want, you should look at other entry points in the API, which you can use to generate a single value at a time in constant space. He thought the vector would be fused away by the library, which is one of the selling points of uvector. Sadly the implementation of uniformArray wasn't done with this purpose in mind. -- Felipe. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Generate random UArray in constant memory space.
Hello Cafe, I tried to generate memory-efficient list of random numbers, so I've used uvector library for this task. But it doesn't work, it still allocates more than 6Gb of memory for the random list of 10 million elements. Here is the code: import Text.Printf import System.Random import Control.Applicative import System.Environment import Data.Array.Vector randomListU :: (Int, Int) - StdGen - Int - (UArr Int) randomListU b g size = unfoldU size gen g where gen g = let (x, g') = randomR b g in JustS (x :*: g') main = do [size] - map read $ getArgs let ints = randomListU (-10, 10) (mkStdGen 1) size printf %d\n (sumU ints) Could someone give a hint, how to implement this function in constant memory space? Thank you in advance. Best regards, Vasyl Pasternak ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Generate random UArray in constant memory space.
Sorry, maybe I should ask more clearer. I've looked at dons article Haskell as fast as C[1], and tried to implement similar algorithm but for list of random numbers. Please look at code: import Text.Printf import Control.Applicative import System.Environment import Data.Array.Vector main = do [size] - map read $ getArgs let ints = enumFromToU 0 size :: UArr Int printf %d\n (sumU ints) This code runs in constant space (on my pc ~25kb allocates on the heap) regardless of array size. So I tried to achieve similar with random list, just to replace `enumFromToU` with my own list generator. So the question - is it possible to implement random list similary to enumFromToU? [1]http://donsbot.wordpress.com/2008/06/04/haskell-as-fast-as-c-working-at-a-high-altitude-for-low-level-performance/ Thank you, Vasyl Pasternak 2010/2/9 Daniel Fischer daniel.is.fisc...@web.de: Am Dienstag 09 Februar 2010 13:18:23 schrieb Vasyl Pasternak: Hello Cafe, I tried to generate memory-efficient list of random numbers, so I've used uvector library for this task. But it doesn't work, it still allocates more than 6Gb of memory for the random list of 10 million elements. Here is the code: Hmm, $ ghc -O2 --make ranVec [1 of 1] Compiling Main ( ranVec.hs, ranVec.o ) Linking ranVec ... $ ./ranVec 1000 +RTS -sstderr 5130 4,919,912,080 bytes allocated in the heap 883,256 bytes copied during GC 26,896 bytes maximum residency (1 sample(s)) 25,620 bytes maximum slop 1 MB total memory in use (0 MB lost due to fragmentation) maximum residency is just eight bytes more than for 100,000 or 1,000,000 numbers. I think that is constant space. The ~5 GB total allocation is sequential, ten million new StdGens are produced and allocated, then immediately garbage collected. I see no problem (except that StdGen is slow, e.g. the Mersenne twister is much faster [and allocates less, but still linear in size]). import Text.Printf import System.Random import Control.Applicative import System.Environment import Data.Array.Vector randomListU :: (Int, Int) - StdGen - Int - (UArr Int) randomListU b g size = unfoldU size gen g where gen g = let (x, g') = randomR b g in JustS (x :*: g') main = do [size] - map read $ getArgs let ints = randomListU (-10, 10) (mkStdGen 1) size printf %d\n (sumU ints) Could someone give a hint, how to implement this function in constant memory space? Thank you in advance. Best regards, Vasyl Pasternak ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Generate random UArray in constant memory space.
Daniel,
Yes, I have 64 bit system.
Maybe you're right. The PRNG code with the same vector size allocates
two times more memory at my PC. (~ 1 Gb)
Thank you,
Vasyl
2010/2/9 Daniel Fischer daniel.is.fisc...@web.de:
Am Tuesday 09 February 2010 19:19:18 schrieben Sie:
Daniel,
I've just run venum2 program locally and here is my results:
$ ./venum2 1000 +RTS -s
./venum2 1000 +RTS -s
500500
22,736 bytes allocated in the heap
688 bytes copied during GC
17,184 bytes maximum residency (1 sample(s))
19,680 bytes maximum slop
1 MB total memory in use (0 MB lost due to fragmentation)
$ ./venum2 10 +RTS -s
./venum2 10 +RTS -s
55
24,152 bytes allocated in the heap
688 bytes copied during GC
17,184 bytes maximum residency (1 sample(s))
19,680 bytes maximum slop
1 MB total memory in use (0 MB lost due to fragmentation)
So my PC shows that there is constant memory allocation. Maybe I'm
doing something wrong ?
Unlikely.
However, I notice that you seem to have a 64-bit system.
I don't know the details, but usually those have more registers than a 32-
bit system, so you can get more complicated loops to run completely in the
registers.
I think that's what happens here, enumFromToU' is just complicated enough
to not run in the registers on my 32-bit system, but still runs in the
registers on your 64-bit system.
The PRNG code is too complicated (too many temporary variables) to run in
the registers on either system (BTW, have you tried it with the specialised
PRNG code in the source file? I'm not sure whether that might be just small
enough to run in the registers on a 64-bit system.).
2010/2/9 Daniel Fischer daniel.is.fisc...@web.de:
Am Tuesday 09 February 2010 15:43:13 schrieben Sie:
Update:
I've implemented `enumFromToU` through `unfoldU`:
enumFromToU' from to = unfoldU (to - from) f from
where f i = let i' = i + 1 in JustS (i' :*: i')
This code behaves similarly to `enumFromToU` (i.e. constantly uses
~25 kb of memory, runs in the same time as above).
Wait,
$ cat venum2.hs
module Main (main) where
import Text.Printf
import Control.Applicative
import System.Environment
import Data.Array.Vector
main = do
[size] - map read $ getArgs
let ints = enumFromToU' 0 size :: UArr Int
printf %d\n (sumU ints)
enumFromToU' from to = unfoldU (to - from) f from
where f i = let i' = i + 1 in JustS (i' :*: i')
$ ghc -O2 --make venum2
[1 of 1] Compiling Main ( venum2.hs, venum2.o )
Linking venum2 ...
$ ./venum2 +RTS -sstderr -RTS 100
./venum2 100 +RTS -sstderr
1784293664
48,256,384 bytes allocated in the heap
6,256 bytes copied during GC
26,804 bytes maximum residency (1 sample(s))
25,524 bytes maximum slop
1 MB total memory in use (0 MB lost due to
fragmentation)
$ ./venum2 +RTS -sstderr -RTS 1000
./venum2 1000 +RTS -sstderr
-2004260032
481,937,552 bytes allocated in the heap
19,516 bytes copied during GC
26,804 bytes maximum residency (1 sample(s))
25,512 bytes maximum slop
1 MB total memory in use (0 MB lost due to
fragmentation)
So we have constant memory, but linear allocation, just like with the
random numbers.
With enumFromToU, also the allocation is constant, so unfoldU
allocates, enumFromToU not.
So the question is why random number list generator eats O(n) memory
?
It doesn't, not here, at least. Using System.Random, the allocation
figures are about ten times as high, but the residency is about the
same. Putting the PRNG code in the same file and tweaking things a bit
(eliminating all intermediate Integers, e.g.), I get
$ cat ran2Vec.hs
{-# LANGUAGE BangPatterns #-}
module Main (main) where
import Text.Printf
import Control.Applicative
import System.Environment
import Data.Array.Vector
randomListU :: (Int, Int) - StdGen - Int - (UArr Int)
randomListU b@(l,h) g size = unfoldU size gen g
where
!k = h-l+1
!b' = 2147483561 `rem` k
gen !g = let (!x, !g') = stdNext g
in JustS ((l+ (x+b') `rem` k) :*: g')
main = do
[size] - map read $ getArgs
let ints = randomListU (-10, 10) (mkStdGen 1) size
printf %d\n (sumU ints)
data StdGen = StdGen {-# UNPACK #-} !Int {-# UNPACK #-} !Int
mkStdGen :: Int - StdGen
mkStdGen s
| s 0 = mkStdGen (-s)
| otherwise = StdGen (s1+1) (s2+1)
where
(q, s1) = s `divMod` 2147483562
s2 = q `mod` 2147483398
{-# INLINE stdNext #-}
stdNext :: StdGen - (Int, StdGen)
-- Returns values in the range stdRange
stdNext (StdGen s1 s2) = z' `seq` g' `seq` (z', g')
where
!g' = StdGen s1'' s2''
!z' = if z 1 then z +
Re: [Haskell-cafe] Generate random UArray in constant memory space.
Bryan, mwc-random is really fast. But it eats to much memory. My previous attempts were to reduce total number of allocations. This package made this possible, but it increases total memory usage. Here is the code: import Text.Printf import System.Random.MWC import Control.Applicative import System.Environment import Data.Array.Vector import Control.Monad.ST randomListU size = runST $ flip uniformArray size = create main = do [size] - map read $ getArgs printf %d\n $ (sumU (randomListU size) :: Int) And sample tests: ./mwcvec 1000 +RTS -s -9198901858466039191 165,312 bytes allocated in the heap 688 bytes copied during GC 17,184 bytes maximum residency (1 sample(s)) 19,680 bytes maximum slop 78 MB total memory in use (1 MB lost due to fragmentation) ./mwcvec 1 +RTS -s 2242701120799374676 165,704 bytes allocated in the heap 688 bytes copied during GC 17,184 bytes maximum residency (1 sample(s)) 19,680 bytes maximum slop 764 MB total memory in use (12 MB lost due to fragmentation) ./mwcvec 10 +RTS -s mwcvec: out of memory (requested 8000634880 bytes) I don't know exactly, but is this a normal behavior ? Thank you, Vasyl 2010/2/9 Bryan O'Sullivan b...@serpentine.com: On Tue, Feb 9, 2010 at 4:18 AM, Vasyl Pasternak vasyl.paster...@gmail.com wrote: I tried to generate memory-efficient list of random numbers, so I've used uvector library for this task. Use the mwc-random package. It provides a function that does exactly this, and produces better quality random numbers with much higher performance (1000x faster) than System.Random or even mersenne-random. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Some thoughts about Hackage
Hi, Yesterday Max complained about documentation for many Haskell modules. But I found another similar problem with Hackage. Before coding some Haskell program I try to find most appropriate libraries, which help me to do task more efficiently. But the problem, that there are to many libraries with similar functionality (for example - networking, web servers etc.). And to find the best solution is not so obvious. My idea is to improve Hackage to help everyone with package selecting. I propose the following: - add download counter for each package, it could show how popular the package is - allow registered users set the quality mark of the package (from 1 to 5) and show the average mark of each packet - add counter which shows how many packages depend on this package (direct/indirect) - create an aggregate package rank (a function on previous three values), similar to Google's PageRank, i. e. rank of the package is proportional to the package mark and weighted rank of dependent packages - allow comments on the package page, so anyone could tell its opinion or other useful info for this package. All messages should be delivered to the maintainer. This is useful, because it could speedup the feedback on the packages, and also could form large knowledge base on each package. Now everyone, who wants to read more about some package should use Google to extract info from HaskellWiki, Haskell-Cafe or hundreds of blog posts from different authors. What do you think ? Best regards, Vasyl Pasternak ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] ANNOUNCE: hs-ffmpeg 0.3.2 - Bindings to FFMpeg library
Hello, I am glad to announce next release of hs-ffmpeg library. Now you could download it from the Hackage along with the ffmpeg-tutorials, which show capabilities of this library. The installation process is a bit tricky now, so welcome to my blog post http://progandprog.blogspot.com/2009/10/video-processing-on-haskell-easy.html, where I am describing how to build and install library. Best regards, Vasyl Pasternak ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Ctrl-C handling in Haskell with curl on Linux
Hi, Yesterday I tried to implement simple tool to download pages, and wanted catch Ctrl-C (and other 'killing' messages) from haskell to handle state saving. Without curl (when I perform some long operation) haskell throws UserInterrupt exception immediately, but if I put long operation, which downloads page from the WEB (from the far-far-away server :) ) than I noticed following issues: - to break my program I have to press Ctrl-C twice - haskell doesn't throw an exception - when I rewrite this code to use signals, haskell, after I press Ctrl-C several times exits with error too many pending signals I put the test code in the end of the letter. Shortly the longTask doesn't handle Ctrl-C and longTask' handles it. I couldn't find any solutions to this problem, I am afraid that this problem could occur in other non-native haskell modules (bindings to C libraries) Many thanks in advance, Vasyl pasternak -- Test code: module Main where import Prelude hiding (catch) import Network.Curl import Control.Exception import Control.Monad import System.IO errorHandler defVal e = do putStrLn $ Error: ++ (show (e :: ErrorCall)) return defVal link = far-far-away-site.com.net getSite curl l = do r - do_curl_ curl l method_GET :: IO (CurlResponse) if respCurlCode r /= CurlOK then error get page failed else return $ respBody r -- this long task doesn't throw user interrupts longTask = do putStrLn Long task started curl - initialize setopts curl [CurlCookieJar cookies] handle (errorHandler ()) $ mapM_ (\_ - getSite curl link return ()) [0..100] return () -- this trows longTask' = do putStrLn long task started let fib n = foldr (*) 1 [1..n] h - openFile /dev/null WriteMode -- never ends mapM_ (hPutStr h . show . fib) [1..] return () onAbort e = do let x = show (e :: AsyncException) putStrLn $ Aborted: ++ x return () main :: IO () main = do handle onAbort longTask putStrLn Exiting ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Ctrl-C handling in Haskell with curl on Linux
Thank you, You give me and idea, and I fixed this annoying bug - we should only wrap all curl code into withCurlDo function, so the longTask function should be following: longTask = do putStrLn Long task started (curl) withCurlDo $ do curl - initialize setopts curl [CurlCookieJar cookies] handle (errorHandler ()) $ mapM_ (\_ - getSite curl link return ()) [0..100] return () Now it works fine and handles interrupts correctly. Best regards, Vasyl Pasternak 2009/9/24 Brandon S. Allbery KF8NH allb...@ece.cmu.edu On Sep 24, 2009, at 06:20 , Brandon S. Allbery KF8NH wrote: On Sep 24, 2009, at 05:30 , Vasyl Pasternak wrote: Yesterday I tried to implement simple tool to download pages, and wanted catch Ctrl-C (and other 'killing' messages) from haskell to handle state saving. Without curl (when I perform some long operation) haskell throws UserInterrupt exception immediately, but if I put long operation, which downloads page from the WEB (from the far-far-away server :) ) than I noticed following issues: You're going to have problems any time a C library installs its own signal handler, which I would expect libcurl to do so it can clean up after itself. This is true even in C-to-C calling; you need a way to hook the signal handler, which some libraries provide in their API and others you just lose. Just occurred to me I should clarify: while most exception handling mechanisms support the concept of re-throwing exceptions to outer exception handlers, POSIX signals do not. The best you could hope for in a library routine which handles signals itself is an API hook into the signal handler; next best is the API returning a signal-occurred error/exception value. Note that I have no idea how the equivalent signaling mechanism works on Win32. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allb...@kf8nh.com system administrator [openafs,heimdal,too many hats] allb...@ece.cmu.edu electrical and computer engineering, carnegie mellon universityKF8NH ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] FFMpeg, SDL and Haskell
Hi all, Last few days I was playing with FFI, FFMpeg and Haskell. Currently I am trying to make this tutorial http://www.dranger.com/ffmpeg/ on Haskell. Now I have done tutorial 01 and tutorial 02 (show video stream in SDL window). The third tutorial is about audio, and I found that audio doesn't supported in SDL bindings completely. So I'd like to fill this gap, but I don't know where to get latest version and to whom I have to send SDL patches. Anyway, does anybody interested in FFMpeg bindings ? Should I put it on Hackage ? The preliminary version I've put on google code: http://hs-ffmpeg.googlecode.com/files/hs-ffmpeg-0.2.0.tar.gz, but this is work on my PC version. Vasyl ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Just 3 = (1+)?
Hi, Haskell expects the function with type (a - m b) in the right side of (=), but you put there function with type (a - a): try: :t (Just 3 =) (Just 3 =) :: (Num a) = (a - Maybe b) - Maybe b and: :t (1+) (1+) :: (Num a) = a - a You should put (1+) into Maybe monad, just do return.(1+), so Just 3 = return . (1+) will return `Just 4` -- Best regards, Vasyl Pasternak 2009/5/9 michael rice nowg...@yahoo.com Why doesn't this work? Michael data Maybe a = Nothing | Just a instance Monad Maybe where return = Just fail = Nothing Nothing = f = Nothing (Just x) = f = f x instance MonadPlus Maybe where mzero = Nothing Nothing `mplus` x = x x `mplus` _ = x [mich...@localhost ~]$ ghci GHCi, version 6.10.1: http://www.haskell.org/ghc/ :? for help Loading package ghc-prim ... linking ... done. Loading package integer ... linking ... done. Loading package base ... linking ... done. Prelude Just 3 = (1+) interactive:1:0: No instance for (Num (Maybe b)) arising from a use of `it' at interactive:1:0-14 Possible fix: add an instance declaration for (Num (Maybe b)) In the first argument of `print', namely `it' In a stmt of a 'do' expression: print it Prelude ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] [ANNOUNCE] hgettext 0.1.10 - last major release
Hello, I've uploaded last (and latest) significant version on hgettext module. Currently it works fine, and has bindings to all gettext functions (from libintl.h). Next versions will be only bug fixes of this version. GNU GetText allows you only to create simple, standalone desktop applications for Linux, to make something complex you should use different trick, fortunately Haskell could do these tricks much easier than other languages (if you want to see, how to build Gtk application, which supports run time language switching, and shows information on two languages simultaneously you could download example from http://hgettext.googlecode.com/files/gtk-hello-0.0.2.tar.gz , or read about it from http://progandprog.blogspot.com/2009/04/multilingual-ui-and-dynamic-language.html) I don't see any strong reasons to write any combinators over this basic bindings. Haskell needs more powerful internationalization library, and I am plan to design it, but it will be completely different from gettext principles, so this library will be released with another name. CHANGELOG: * Added '--version' command line parameter to hgettext tool * Added support for literate haskell files to hgettext tool * Changed all functions to throw IOError when they fails instead of return Nothing * Added bindings for: ngettext, dgettext, dngettext, dcgettext and dcngettext functions -- Best regards, Vasyl Pasternak ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] ANNOUNCE hgettext-0.1.5 - GetText based internationalization of Haskell programs
Hello, I have extended my previous version of the library to support distribution and installation of PO files. Source tarball - http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hgettext Also I described how to use this feature to distribute haskell packages in my blog entry http://progandprog.blogspot.com/2009/04/configure-and-install-internationalized.html Same description was added to the Haskell Wiki - http://www.haskell.org/haskellwiki/Internationalization_of_Haskell_programs Complete example, which uses internationalization capabilities is here: http://hgettext.googlecode.com/files/hello-0.1.3.tar.gz --- Best regards, Vasyl Pasternak ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Internationalization of Haskell code
In my blog http://progandprog.blogspot.com/2009/03/i18n-and-haskell.html, I've described process of internationalization Haskell programms using GNU GetText. I wrote hgettext tool and Text.I18N.GetText library to help with internationalization (latest sources on http://hgettext.googlecode.com/files/hgettext-0.1.2.tar.gz). A bit later I plan to put it to hackage and wiki (waiting for account). -- Best regards, Vasyl Pasternak ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Language.Haskell.Parser question
Hi, I want to parse haskell file to find all calls to function 'foo' and gathers a create a list of all argumets, which passed to it. E.g. from the following code: f1 = foo 5 f2 = foo 8 f3 = foo 9 I want to extract a list [5, 8, 9] (suppouse function takes only one argument) The most obvious way is to use Language.Haskell for this task. The parser works pretty good, but its output data type is terrible. As I understand, I need to extract all objects that looks like HsApp (HsVar (UnQual (HsIdent foo))) The question is, is there a method to do it quickly or I have to process each object of different type separately ? Maybe it is a work for a template Haskell ? Thanks. -- Best regards, Vasyl Pasternak ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Language.Haskell.Parser question
Wow, uniplate is the library from my dreams :) I knew there should be easy, elegant and simple solution (I hate brute force, that's why I start using Haskell). Many thanks. 2009/3/26 Neil Mitchell ndmitch...@gmail.com: Hi f1 = foo 5 f2 = foo 8 f3 = foo 9 I want to extract a list [5, 8, 9] (suppouse function takes only one argument) Firstly, use haskell-src-exts and Language.Haskell.Exts - its a much better library, deals with many extensions, and gives you everything Language.Haskell did. parser works pretty good, but its output data type is terrible. As I understand, I need to extract all objects that looks like HsApp (HsVar (UnQual (HsIdent foo))) It's trivial, if you use a generics solution, say Uniplate: http://community.haskell.org/~ndm/uniplate [x | App foo x - universeBi src, prettyPrint foo == foo ] I often use prettyPrint to follow down Ident/Qual/UnQual paths without having to know as much of the data type. Using universeBi makes it easy to find everything that occurs everywhere. If you attempt this without using a generics library, I'd say you are destined to fail. Thanks Neil -- Best regards, Vasyl Pasternak ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ACM Task for C++ and Java programmers in Haskell. How to make code faster?
I changed the program, now it similar to the program from the wiki (http://www.haskell.org/haskellwiki/Phone_number) The version with ByteString compared to version with ordinary Strings works 3.5 times faster. (I put it to http://hpaste.org/fastcgi/hpaste.fcgi/view?id=2830) But version with Data.Trie dissapointed me, it works 5 times slover than version with Data.Map ByteString. (here is the code http://hpaste.org/fastcgi/hpaste.fcgi/view?id=2829) Anyway, thanks to everyone who helped me, Haskell is really powerfull tool in clever hands :) 2009/3/23 wren ng thornton w...@freegeek.org: Vasyl Pasternak wrote: The entire code I placed on http://hpaste.org/fastcgi/hpaste.fcgi/view?id=2764 Could someone help me to make this code faster? I'd like to see solution that will be elegant and fast, without heavy optimizations, that will make code unreadable. Also, if it possible, prepare the program to support SMP parallelism. The solution's already been posted, but to make this particular code faster, I recommend using Data.Trie instead of Data.Map ByteString. Tries are faster for lookup since they don't redundantly check the prefix of the query; also they're better for memory usage because they don't store redundant copies of the prefixes. http://hackage.haskell.org/cgi-bin/hackage-scripts/package/bytestring-trie -- Live well, ~wren ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe -- Best regards, Vasyl Pasternak ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] ACM Task for C++ and Java programmers in Haskell. How to make code faster?
Hi! Recently I've found interesting ACM research on C++ and Java efficiency. This task also was been solved on lisp/scheme and is described on http://www.flownet.com/ron/papers/lisp-java/ I tried to solve this task on Haskell but stuck with efficiency problems (approx in 1000 times slower and takes 20 times more memory). I hope the approach, I've used is correct and problem with data structures and implementation. It took more than 81% of time for garbage collection. I tried to use ByteStrings instead of Strings, but unfortunately it didn't help. The entire code I placed on http://hpaste.org/fastcgi/hpaste.fcgi/view?id=2764 The original instructions are on: http://www.flownet.com/ron/papers/lisp-java/instructions.html Dictionary file could be downloaded from: http://www.flownet.com/ron/papers/lisp-java/dictionary. Input data: http://www.flownet.com/ron/papers/lisp-java/input.txt Expected output: http://www.flownet.com/ron/papers/lisp-java/output.txt Could someone help me to make this code faster? I'd like to see solution that will be elegant and fast, without heavy optimizations, that will make code unreadable. Also, if it possible, prepare the program to support SMP parallelism. Very thanks in advance. Hope you'll enjoy this task :) -- Best regards, Vasyl Pasternak ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ACM Task for C++ and Java programmers in Haskell. How to make code faster?
Thanks for the point. I've found haskell solution there http://www.haskell.org/haskellwiki/Phone_number The second (shorter) solution is fantastic, it beats all other languages, but I'll never wrote such code in reasonable time :) 2009/3/22 Bulat Ziganshin bulat.zigans...@gmail.com: Hello Vasyl, Sunday, March 22, 2009, 8:25:02 PM, you wrote: i believe that i already seen this problem here a few years ago :) what search structure you was used? i think that immutable hash (represented as array of lists) would be useful here Hi! Recently I've found interesting ACM research on C++ and Java efficiency. This task also was been solved on lisp/scheme and is described on http://www.flownet.com/ron/papers/lisp-java/ I tried to solve this task on Haskell but stuck with efficiency problems (approx in 1000 times slower and takes 20 times more memory). I hope the approach, I've used is correct and problem with data structures and implementation. It took more than 81% of time for garbage collection. I tried to use ByteStrings instead of Strings, but unfortunately it didn't help. The entire code I placed on http://hpaste.org/fastcgi/hpaste.fcgi/view?id=2764 The original instructions are on: http://www.flownet.com/ron/papers/lisp-java/instructions.html Dictionary file could be downloaded from: http://www.flownet.com/ron/papers/lisp-java/dictionary. Input data: http://www.flownet.com/ron/papers/lisp-java/input.txt Expected output: http://www.flownet.com/ron/papers/lisp-java/output.txt Could someone help me to make this code faster? I'd like to see solution that will be elegant and fast, without heavy optimizations, that will make code unreadable. Also, if it possible, prepare the program to support SMP parallelism. Very thanks in advance. Hope you'll enjoy this task :) -- Best regards, Bulatmailto:bulat.zigans...@gmail.com -- Best regards, Vasyl Pasternak ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
