max heap such that the largest operation
> fits into 70% of the max heap. Additionally, memory configurations also
> impact operator selection - for example, we only compile broadcast-based
> matrix multiplications if the smaller input fits twice in the driver and in
> the broadcast budget of
r row, you
>> could store it as a column vector with FK2 = rowIndexMax(FK) and
>> subsequently reconstruct it via FK = table(seq(1,nrow(FK2)), FK2,
>> nrow(FK2), N), for which we will compile a dedicated operator that does row
>> expansions. You don't necessarily need the last
Hi all,
I was playing with a super sparse matrix FK, 2e7 by 1e6, with only one
non-zero value on each row, that is 2e7 non-zero values in total.
With driver memory of 1GB and executor memory of 100GB, I found the HOP
"Spark chkpoint", which is used to pin the FK matrix in memory, is really
ould
> significantly reduce the memory pressure.
>
> Regards,
> Matthias
>
> On 5/3/2017 9:38 AM, Mingyang Wang wrote:
> > Hi all,
> >
> > I was playing with a super sparse matrix FK, 2e7 by 1e6, with only one
> > non-zero value on each row, that is 2e7 n
-- 3) + 0.000 sec 1
-- 4) print 0.000 sec 1
-- 5) rmvar 0.000 sec 5
-- 6) createvar 0.000 sec 1
-- 7) assignvar 0.000 sec 1
-- 8) cpvar 0.000 sec 1
Regards,
Mingyang
On Thu, May 4, 2017 at 9:48 PM Mingyang Wang <miw...@eng.ucsd.edu> wrote:
> Hi Matthias,
>
> Thanks for the patch
detailed setup again (e.g., num executors, data distribution, etc). Thanks.
>
> Regards,
> Matthias
>
>
> On Thu, May 4, 2017 at 9:55 PM, Mingyang Wang <miw...@eng.ucsd.edu> wrote:
>
> > Out of curiosity, I increased the driver memory to 10GB, and then all
>
