[Haskell-cafe] (Repa) hFlush: illegal operation

2011-08-24 Thread Michael Orlitzky 
I'm using Repa to process a ton of MRI data. The basic process is,

  * Read in the data
  * Create a big 'ol data structure (grid) from it
  * Compute the output in parallel using 'traverse'
  * Write the output to file

However, during the last step, I'm getting,

  $ ./bin/spline3 +RTS -N4
  spline3: output.txt: hFlush: illegal operation (handle is closed)

Here's the entirety of my Main.hs:

 module Main
 where

 import Data.Array.Repa (
   DIM3,
   Z(..),
   (:.)(..),
   )

 import Values
 import Grid(make_grid, zoom)

 mri_shape :: DIM3
 mri_shape = (Z :. 256 :. 256 :. 109)

 main :: IO ()
 main = do
   mridata - read_values_3d mri_shape ./data/mridata.txt
   let g = make_grid 1 mridata
   let output = zoom g 1
   write_values_1d output output.txt


And the two functions used from the Values module,

 type Values1D = Array DIM1 Double
 type Values3D = Array DIM3 Double

 read_values_1d :: FilePath - IO Values1D
 read_values_1d path = readVectorFromTextFile path


 read_values_3d :: DIM3 - FilePath - IO Values3D
 read_values_3d sh path = do
   one_d - read_values_1d path
   return $ reshape sh one_d


 write_values_1d :: Values3D - FilePath - IO ()
 write_values_1d v3d path = do
   let size3d = size $ extent v3d
   let shape1d = (Z :. size3d)
   let v1d = reshape shape1d v3d
   writeVectorToTextFile v1d path


Am I doing something obviously wrong here? It takes about 45 minutes to
run up to the point of failure, so I haven't attempted much trial  error.

Packages:

  * ghc-7.2.1
  * repa-2.1.1.6
  * repa-io-2.1.0.1


___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Re: [Haskell-cafe] (Repa) hFlush: illegal operation

2011-08-24 Thread Ben Lippmeier 

On 25/08/2011, at 7:15 , Michael Orlitzky wrote:

 I'm using Repa to process a ton of MRI data. The basic process is,
 
  * Read in the data
  * Create a big 'ol data structure (grid) from it
  * Compute the output in parallel using 'traverse'
  * Write the output to file
 
 However, during the last step, I'm getting,
 
  $ ./bin/spline3 +RTS -N4
  spline3: output.txt: hFlush: illegal operation (handle is closed)


 read_values_1d :: FilePath - IO Values1D
 read_values_1d path = readVectorFromTextFile path

The implementation of the text IO functions is fairly naive, just using Haskell 
Strings etc under the covers. It may have problems with massive files. 

Can you send me some gzipped data of the same size/form as what you're using, 
or tell me where to download it? Even if your real source data is several GB in 
size, if you make a test file with mostly zeros it should gzip down to nothing.

Also, what is the native form of the data you are using? If it's in some 
standard binary form it may just be easier to write a native Repa loader for it.

Cheers,
Ben.


___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe