Same here on a 12gb ram machine
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.5.0-dev+429 (2015-09-29 09:47 UTC)
_/ |\__'_|_|_|\__'_| | Commit f71e449 (14 days old master)
|__/ | x86_64-w64-mingw32
julia> using DataFrames
julia> train = readtable("./test.csv");
ERROR: OutOfMemoryError()
in resize! at array.jl:452
in readnrows! at
C:\Users\Mustafa\.julia\v0.5\DataFrames\src\dataframe\io.jl:164
in readtable! at
C:\Users\Mustafa\.julia\v0.5\DataFrames\src\dataframe\io.jl:767
in readtable at
C:\Users\Mustafa\.julia\v0.5\DataFrames\src\dataframe\io.jl:847
in readtable at
C:\Users\Mustafa\.julia\v0.5\DataFrames\src\dataframe\io.jl:893
On Tuesday, October 13, 2015 at 3:47:58 PM UTC-4, Yichao Yu wrote:
>
>
> On Oct 13, 2015 2:47 PM, "Grey Marsh" <[email protected] <javascript:>>
> wrote:
>
> Which julia version are you using. There's sime gc tweak on 0.4 for that.
>
> >
> > I was trying to load the training dataset from springleaf marketing
> response on Kaggle. The csv is 921 mb, has 145321 row and 1934 columns. My
> machine has 8 gb ram and julia ate 5.8gb+ memory after that I stopped julia
> as there was barely any memory left for OS to function properly. It took
> about 5-6 minutes later for the incomplete operation. I've windows 8
> 64bit. Used the following code to read the csv to Julia:
> >
> > using DataFrames
> > train = readtable("C:\\train.csv")
> >
> > Next I tried to to load the same file in python:
> >
> > import pandas as pd
> > train = pd.read_csv("C:\\train.csv")
> >
> > This took ~2.4gb memory, about a minute time
> >
> > Checking the same in R again:
> > df = read.csv('E:/Libraries/train.csv', as.is = T)
> >
> > This took 2-3 minutes and consumes 3.5gb mem on the same machine.
> >
> > Why such discrepancy and why Julia even fails to load the csv before
> running out of memory? If there is any better way to get the file loaded in
> Julia?
> >
> >
>
