jobs much slower in cluster mode vs local

[Saif.A.Ellafi]

Hello,

In general, I am usually able to run spark submit jobs in local mode, in a 
32-cores node with plenty of memory ram. The performance is significantly 
faster in local mode than when using a cluster of spark workers.

How can this be explained and what measures can one take in order to improve 
such performance?
Usually a job that takes 35 seconds in local mode takes around 48 seconds in a 
small cluster.

Thanks,
Saif

Re: jobs much slower in cluster mode vs local

[Jiří Syrový]

Hi,

you can try to use spark job server and submit jobs to it. The thing is
that the most expensive part is context creation.

J.

2016-01-15 15:28 GMT+01:00 :

> Hello,
>
> In general, I am usually able to run spark submit jobs in local mode, in a
> 32-cores node with plenty of memory ram. The performance is significantly
> faster in local mode than when using a cluster of spark workers.
>
> How can this be explained and what measures can one take in order to
> improve such performance?
> Usually a job that takes 35 seconds in local mode takes around 48 seconds
> in a small cluster.
>
> Thanks,
> Saif
>
>

RE: jobs much slower in cluster mode vs local

[Spencer, Alex (Santander)]

That's not that much of a difference given the overhead of cluster management. 
I would have thought a job should take minutes before you'll see a performance 
improvement on using cluster mode?

Kind Regards,
Alex.

From: saif.a.ell...@wellsfargo.com [mailto:saif.a.ell...@wellsfargo.com]
Sent: 15 January 2016 14:29
To: user@spark.apache.org
Subject: jobs much slower in cluster mode vs local

Hello,

In general, I am usually able to run spark submit jobs in local mode, in a 
32-cores node with plenty of memory ram. The performance is significantly 
faster in local mode than when using a cluster of spark workers.

How can this be explained and what measures can one take in order to improve 
such performance?
Usually a job that takes 35 seconds in local mode takes around 48 seconds in a 
small cluster.

Thanks,
Saif

Emails aren't always secure, and they may be intercepted or changed after
they've been sent. Santander doesn't accept liability if this happens. If you
think someone may have interfered with this email, please get in touch with the
sender another way. This message doesn't create or change any contract.
Santander doesn't accept responsibility for damage caused by any viruses
contained in this email or its attachments. Emails may be monitored. If you've
received this email by mistake, please let the sender know at once that it's
gone to the wrong person and then destroy it without copying, using, or telling
anyone about its contents.
Santander UK plc Reg. No. 2294747 and Abbey National Treasury Services plc Reg.
No. 2338548 Registered Offices: 2 Triton Square, Regent's Place, London NW1 3AN.
Registered in England. www.santander.co.uk. Authorised by the Prudential
Regulation Authority and regulated by the Financial Conduct Authority and the
Prudential Regulation Authority. FCA Reg. No. 106054 and 146003 respectively.
Santander Sharedealing is a trading name of Abbey Stockbrokers Limited Reg. No.
02666793. Registered Office: Kingfisher House, Radford Way, Billericay, Essex
CM12 0GZ. Authorised and regulated by the Financial Conduct Authority. FCA Reg.
No. 154210. You can check this on the Financial Services Register by visiting
the FCA’s website www.fca.org.uk/register or by contacting the FCA on 0800 111
6768. Santander UK plc is also licensed by the Financial Supervision Commission
of the Isle of Man for its branch in the Isle of Man. Deposits held with the
Isle of Man branch are covered by the Isle of Man Depositors’ Compensation
Scheme as set out in the Isle of Man Depositors’ Compensation Scheme Regulations
2010. In the Isle of Man, Santander UK plc’s principal place of business is at
19/21 Prospect Hill, Douglas, Isle of Man, IM1 1ET. Santander and the flame logo
are registered trademarks.
Santander Asset Finance plc. Reg. No. 1533123. Registered Office: 2 Triton
Square, Regent’s Place, London NW1 3AN. Registered in England. Santander
Corporate & Commercial is a brand name used by Santander UK plc, Abbey National
Treasury Services plc and Santander Asset Finance plc.
Ref:[PDB#1-4A]
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

RE: jobs much slower in cluster mode vs local

[Saif.A.Ellafi]

Thank you, this looks useful indeed for what I have in mind.

Saif

From: Jiří Syrový [mailto:syrovy.j...@gmail.com]
Sent: Friday, January 15, 2016 12:06 PM
To: Ellafi, Saif A.
Cc: user@spark.apache.org
Subject: Re: jobs much slower in cluster mode vs local

Hi,

you can try to use spark job server and submit jobs to it. The thing is that 
the most expensive part is context creation.
J.

2016-01-15 15:28 GMT+01:00 
<saif.a.ell...@wellsfargo.com<mailto:saif.a.ell...@wellsfargo.com>>:
Hello,

In general, I am usually able to run spark submit jobs in local mode, in a 
32-cores node with plenty of memory ram. The performance is significantly 
faster in local mode than when using a cluster of spark workers.

How can this be explained and what measures can one take in order to improve 
such performance?
Usually a job that takes 35 seconds in local mode takes around 48 seconds in a 
small cluster.

Thanks,
Saif