Hi,
I'm testing Beam on Spark.
I use spark example code WordCount processing 1G data file, cost 1 minutes.
However, I use Beam example code WordCount processing the same file, cost
30minutes.
My Spark parameter is : --deploy-mode client --executor-memory 1g
--num-executors 1 --driver-memory 1g
My Spark version is 2.3.1, Beam version is 2.5
Is there any optimization method?
Thank you.- How to optimize the performance of Beam on Spark 段丁瑞
- Re: How to optimize the performance of Beam on ... Robert Bradshaw
- Re: How to optimize the performance of Beam... Tim Robertson
- Re: How to optimize the performance of Beam on ... Jean-Baptiste Onofré
- Re: How to optimize the performance of Beam... Jean-Baptiste Onofré
- Re: How to optimize the performance of ... Jean-Baptiste Onofré
- Re: Re: How to optimize the perform... 段丁瑞
- Re: Re: How to optimize the pe... Tim Robertson
- Re: Re: How to optimize th... 段丁瑞
- Re: Re: How to optimiz... Robert Bradshaw
- Re: Re: How to optimiz... Tim Robertson
