Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?

[Paul Analyst]

one vector 10^9 it is about 16 GB,
Paul

W dniu 2015-01-07 o 13:13, Tim Holy pisze:

Did you try calculating how much memory the right hand side of that assignment
requires? (Remember that sizeof(Float64) == 8).

--Tim

On Wednesday, January 07, 2015 03:25:54 AM paul analyst wrote:

    _       _ _(_)_     |  A fresh approach to technical computing
   (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
    _ _   _| |_  __ _   |  Type "help()" for help.

   | | | | | | |/ _` |  |
   | | |
   | | |_| | | | (_| |  |  Version 0.3.3 (2014-10-21 20:18 UTC)

  _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org release

|__/                   |  i686-linux-gnu

....

dset[1:10^8,1]=rand(10^8);

julia> size(dset)
(1000000000,10000000)

julia> dset[1:10^9,1]=rand(10^9);
ERROR: invalid Array size
  in rand at random.jl:123

julia>

Julia now is using 32% of ram, 2.5 GB of 7.8 GB aviable..
Ubuntu 14.10
Why ?  free size of HD is now 30GB
Paul


W dniu wtorek, 6 stycznia 2015 21:04:53 UTC+1 użytkownik paul analyst

napisał:

2.if sum k and l > 9 Julia cant work. Is it to big size for hdf5 or for

system (Win7 64 Home Premium) ?

Not sure. It works for me (Kubuntu Linux 14.04).
I checked on Ubuntu :

If sum of "k" or/and  "l" is more then 7 I have problem with reading
vectors (cols)

samsung2@samsung2:~$ julia

                _

_ _ _(_)_ | A fresh approach to technical computing (_) | (_) (_) | Documentation: http://docs.julialang.org _ _ _| |_ __ _ | Type "help()" for help. | | | | | | |/ _` | |

   | | |
   | | |_| | | | (_| |  |  Version 0.3.3 (2014-10-21 20:18 UTC)

_/ |\__'_|_|_|\__'_| | Official http://julialang.org release |__/ | i686-linux-gnu

julia> using HDF5

julia> hfi=h5open("F_big.h5","w");close(hfi)

julia> k,l=8,8    ;

julia> fid = h5open("F_big.h5","r+")
HDF5 data file: F_big.h5

julia> fid["mygroup/A"]=rand(2)#niepotrzebny

2-element Array{Float64,1}:
  0.318459
  0.258055

julia> g = fid["mygroup"]
HDF5 group: /mygroup (file: F_big.h5)

julia> dset = d_create(g, "F", datatype(Float64), dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)

julia> h5read("F_big.h5","mygroup/F",(:,1))

100000000x1 Array{Float64,2}:
  0.0
  0.0
  0.0
  0.0
  0.0
  0.0
  0.0
  0.0
  0.0
  0.0
  ⋮
  0.0
  0.0
  0.0
  0.0
  0.0
  0.0
  0.0
  0.0
  0.0

julia> close(fid)

julia> hfi=h5open("F_big.h5","w");close(hfi)

julia> k,l=9,8    ;

julia> fid = h5open("F_big.h5","r+")
HDF5 data file: F_big.h5

julia> fid["mygroup/A"]=rand(2)#niepotrzebny

2-element Array{Float64,1}:
  0.129214
  0.4785

julia> g = fid["mygroup"]
HDF5 group: /mygroup (file: F_big.h5)

julia> dset = d_create(g, "F", datatype(Float64), dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)

julia> h5read("F_big.h5","mygroup/F",(:,1))
ERROR: invalid Array size

  in Array at base.jl:223
  in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
  in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
  in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
  in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602

julia> close(fid)

julia> h5read("F_big.h5","mygroup/F",(1:2,:))

2x100000000 Array{Float64,2}:
  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0

0.0

  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0

0.0

julia> h5read("F_big.h5","mygroup/F",(:,1))
ERROR: invalid Array size

  in Array at base.jl:223
  in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
  in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
  in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
  in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602

julia>

Paul