Re: [julia-users] R code to Julia Performance issues with pmap

[Michael Krabbe Borregaard]

Ah, forgot that the system.time is a bit tricky when running parallel cores

Re: [julia-users] R code to Julia Performance issues with pmap

[jascisen]

This is not entirely true in R:

Details:

 ‘proc.time’ returns five elements for backwards compatibility, but
 its ‘print’ method prints a named vector of length 3.  The first
 two entries are the total user and system CPU times of the current
 R process and any child processes on which it has waited, and the
 third entry is the ‘real’ elapsed time since the process was
 started.


On Wednesday, July 6, 2016 at 5:11:04 PM UTC+2, Michael Borregaard wrote:
>
> I am not seeing your speed-up in R? elapsed is less time, but user 
> significantly more, and it is the sum that counts.
> When executing in parallel the language needs to copy the data to the 
> workers. If the matrices are large, that takes longer than the speedup of 
> the parallel execution. See what happens with a smaller matrix and then 
> repeating the operation on the workers.
>
>

Re: [julia-users] R code to Julia Performance issues with pmap

[Michael Borregaard]

I am not seeing your speed-up in R? elapsed is less time, but user 
significantly more, and it is the sum that counts.
When executing in parallel the language needs to copy the data to the 
workers. If the matrices are large, that takes longer than the speedup of 
the parallel execution. See what happens with a smaller matrix and then 
repeating the operation on the workers.

Re: [julia-users] R code to Julia Performance issues with pmap

[jascisen]

Yes, but I am not using BLAS or FFT transforms so it si a bit surprising 
that I am not getting any speed improvements

On Wednesday, July 6, 2016 at 2:17:49 AM UTC+2, Stefan Karpinski wrote:
>
> Similar question and answer: 
> http://stackoverflow.com/questions/38075163/julia-uses-only-20-30-of-my-cpu-what-should-i-do/38075939
> .
>
> On Tue, Jul 5, 2016 at 11:26 AM,  wrote:
>
>> I am a complete newcomer to Julia and trying to port some of my R code to 
>> it;
>> Basically I have rewritten the following R code in Julia:
>>
>> library(parallel)
>> 
>> eps_1<-rnorm(100)
>> eps_2<-rnorm(100)
>> 
>> large_matrix<-ifelse(cbind(eps_1,eps_2)>0,1,0)
>> matrix_to_compare = expand.grid(c(0,1),c(0,1))
>> indices<-seq(1,100,4)
>> large_matrix<-lapply(indices,function(i)(large_matrix[i:(i+3),]))
>> 
>> function_compare<-function(x){
>>   which((rowSums(x==matrix_to_compare)==2) %in% TRUE)
>> }
>> 
>> > system.time(lapply(large_matrix,function_compare))
>>user  system elapsed 
>>  38.812   0.024  38.828 
>> > system.time(mclapply(large_matrix,function_compare,mc.cores=11))
>>user  system elapsed 
>>  63.128   1.648   6.108 
>>
>> As one can notice I am getting significant speed-up when going from one 
>> core to 11. Now I am trying to do the same in Julia:
>>
>> using Distributions;
>> @everywhere using Iterators;
>> d = Normal();
>> 
>> eps_1 = rand(d,100);
>> eps_2 = rand(d,100);
>> 
>> #Define cluster:
>> addprocs(11);
>> 
>> #Create a large matrix:
>> large_matrix = hcat(eps_1,eps_2).>=0;
>> indices = collect(1:4:100)
>> 
>> #Split large matrix:
>> large_matrix = [large_matrix[i:(i+3),:] for i in indices];
>> 
>> #Define the function to apply:
>> @everywhere function function_split(x)
>> matrix_to_compare = 
>> transpose(reinterpret(Int,collect(product([0,1],[0,1])),(2,4)));
>> matrix_to_compare = matrix_to_compare.>0;
>> find(sum(x.==matrix_to_compare,2).==2)
>> end
>> 
>> @time map(function_split,large_matrix )
>> @time pmap(function_split,large_matrix )
>>  5.167820 seconds (22.00 M allocations: 2.899 GB, 12.83% gc time)
>>  18.569198 seconds (40.34 M allocations: 2.082 GB, 5.71% gc time)
>>
>> I somehow do not understand why parallel map function does not work for 
>> me. Maybe somebody can point me to a correct solution. 
>>
>
>

Re: [julia-users] R code to Julia Performance issues with pmap

[Stefan Karpinski]

Similar question and answer:
http://stackoverflow.com/questions/38075163/julia-uses-only-20-30-of-my-cpu-what-should-i-do/38075939
.

On Tue, Jul 5, 2016 at 11:26 AM,  wrote:

> I am a complete newcomer to Julia and trying to port some of my R code to
> it;
> Basically I have rewritten the following R code in Julia:
>
> library(parallel)
>
> eps_1<-rnorm(100)
> eps_2<-rnorm(100)
>
> large_matrix<-ifelse(cbind(eps_1,eps_2)>0,1,0)
> matrix_to_compare = expand.grid(c(0,1),c(0,1))
> indices<-seq(1,100,4)
> large_matrix<-lapply(indices,function(i)(large_matrix[i:(i+3),]))
>
> function_compare<-function(x){
>   which((rowSums(x==matrix_to_compare)==2) %in% TRUE)
> }
>
> > system.time(lapply(large_matrix,function_compare))
>user  system elapsed
>  38.812   0.024  38.828
> > system.time(mclapply(large_matrix,function_compare,mc.cores=11))
>user  system elapsed
>  63.128   1.648   6.108
>
> As one can notice I am getting significant speed-up when going from one
> core to 11. Now I am trying to do the same in Julia:
>
> using Distributions;
> @everywhere using Iterators;
> d = Normal();
>
> eps_1 = rand(d,100);
> eps_2 = rand(d,100);
>
> #Define cluster:
> addprocs(11);
>
> #Create a large matrix:
> large_matrix = hcat(eps_1,eps_2).>=0;
> indices = collect(1:4:100)
>
> #Split large matrix:
> large_matrix = [large_matrix[i:(i+3),:] for i in indices];
>
> #Define the function to apply:
> @everywhere function function_split(x)
> matrix_to_compare =
> transpose(reinterpret(Int,collect(product([0,1],[0,1])),(2,4)));
> matrix_to_compare = matrix_to_compare.>0;
> find(sum(x.==matrix_to_compare,2).==2)
> end
>
> @time map(function_split,large_matrix )
> @time pmap(function_split,large_matrix )
>  5.167820 seconds (22.00 M allocations: 2.899 GB, 12.83% gc time)
>  18.569198 seconds (40.34 M allocations: 2.082 GB, 5.71% gc time)
>
> I somehow do not understand why parallel map function does not work for
> me. Maybe somebody can point me to a correct solution.
>

[julia-users] R code to Julia Performance issues with pmap

[jascisen]

I am a complete newcomer to Julia and trying to port some of my R code to 
it;
Basically I have rewritten the following R code in Julia:

library(parallel)

eps_1<-rnorm(100)
eps_2<-rnorm(100)

large_matrix<-ifelse(cbind(eps_1,eps_2)>0,1,0)
matrix_to_compare = expand.grid(c(0,1),c(0,1))
indices<-seq(1,100,4)
large_matrix<-lapply(indices,function(i)(large_matrix[i:(i+3),]))

function_compare<-function(x){
  which((rowSums(x==matrix_to_compare)==2) %in% TRUE)
}

> system.time(lapply(large_matrix,function_compare))
   user  system elapsed 
 38.812   0.024  38.828 
> system.time(mclapply(large_matrix,function_compare,mc.cores=11))
   user  system elapsed 
 63.128   1.648   6.108 

As one can notice I am getting significant speed-up when going from one 
core to 11. Now I am trying to do the same in Julia:

using Distributions;
@everywhere using Iterators;
d = Normal();

eps_1 = rand(d,100);
eps_2 = rand(d,100);

#Define cluster:
addprocs(11);

#Create a large matrix:
large_matrix = hcat(eps_1,eps_2).>=0;
indices = collect(1:4:100)

#Split large matrix:
large_matrix = [large_matrix[i:(i+3),:] for i in indices];

#Define the function to apply:
@everywhere function function_split(x)
matrix_to_compare = 
transpose(reinterpret(Int,collect(product([0,1],[0,1])),(2,4)));
matrix_to_compare = matrix_to_compare.>0;
find(sum(x.==matrix_to_compare,2).==2)
end

@time map(function_split,large_matrix )
@time pmap(function_split,large_matrix )
 5.167820 seconds (22.00 M allocations: 2.899 GB, 12.83% gc time)
 18.569198 seconds (40.34 M allocations: 2.082 GB, 5.71% gc time)

I somehow do not understand why parallel map function does not work for me. 
Maybe somebody can point me to a correct solution.