[ https://issues.apache.org/jira/browse/SPARK-25552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-25552. ------------------------------- Resolution: Invalid This is too broad. Literally 10000 things change from 1.6 to 2.3. You'd have to at least narrow this down to what's taking so much memory. > Upgrade from Spark 1.6.3 to 2.3.0 seems to make jobs use about 50% more memory > ------------------------------------------------------------------------------ > > Key: SPARK-25552 > URL: https://issues.apache.org/jira/browse/SPARK-25552 > Project: Spark > Issue Type: Bug > Components: Java API, Spark Core > Affects Versions: 2.3.0 > Environment: Originally found in an AWS Kubernetes environment with > Spark Embedded. > Also happens in a small scale with Spark Embedded both in Linux and MacOS. > Reporter: Nuno Azevedo > Priority: Major > Attachments: Spark1.6-50GB.png, Spark2.3-50GB.png, Spark2.3-70GB.png > > > After upgrading from Spark 1.6.3 to 2.3.0 our jobs started to need about 50% > more memory to run. The Spark properties used were the defaults in both > versions. > > For instance, before we were running a job with Spark 1.6.3 and it was > running fine with 50 GB of memory. > !Spark1.6-50GB.png|width=800,height=456! > > After upgrading to Spark 2.3.0, when running the same job again with the same > 50 GB of memory it failed due to out of memory. > !Spark2.3-50GB.png|width=800,height=366! > > Then, we started incrementing the memory until we were able to run the job, > which was with 70 GB. > !Spark2.3-70GB.png|width=800,height=366! > > The Spark upgrade was the only change in our environment. After taking a look > at what seems to be causing this we noticed that Kryo Serializer is the main > culprit for the raise in memory consumption. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org