Spark uses a minimum of 2g per executor with no data or work. There is always one executor and one driver with Spark. Welcome to Big Data.
A 4g server is not sufficient to run Spark let alone the rest of the PIO stack. 16g minimum is my recommendation—and I do mean minimum. Machine learning is not an application you can squeeze into small RAM machines. On Dec 13, 2017, at 4:14 AM, Noelia Osés Fernández <[email protected]> wrote: I should mention that when I reboot the server and before I start PIO 3GB of RAM are already used, so there is only 1GB free RAM for PIO. On 13 December 2017 at 13:08, Noelia Osés Fernández <[email protected] <mailto:[email protected]>> wrote: Hi all, I have a cloud server with 4GB RAM in which I have installed PIO 0.12.0-incubating and a few other things. However, I'm having trouble running even the smallest examples as it runs out of RAM (plus the swap is also full). I have set a limit for ES in jvm.options as follows: -Xms512m -Xmx512m Before training and deploying I limit java's RAM usage as follows: export JAVA_OPTS="-Xmx1g -Xms1g -Dfile.encoding=UTF-8" pio train -- --driver-memory 1G --executor-memory 1G nohup pio deploy > deploy.out & Are there any more measures I can take to limit RAM usage? I would like to know I've done absolutely everything possible before paying to get more RAM on the server. Thank you for your help! Noelia -- <http://www.vicomtech.org/> Noelia Osés Fernández, PhD Senior Researcher | Investigadora Senior [email protected] <mailto:[email protected]> +[34] 943 30 92 30 Data Intelligence for Energy and Industrial Processes | Inteligencia de Datos para Energía y Procesos Industriales <https://www.linkedin.com/company/vicomtech> <https://www.youtube.com/user/VICOMTech> <https://twitter.com/@Vicomtech_IK4> member of: <http://www.graphicsmedia.net/> <http://www.ik4.es/> Legal Notice - Privacy policy <http://www.vicomtech.org/en/proteccion-datos>
