[ https://issues.apache.org/jira/browse/SPARK-9539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14651073#comment-14651073 ]
Sean Owen commented on SPARK-9539: ---------------------------------- This just shows Spark is using memory. It's normal to use some of the permanent generation. Your jstat dump shows normal growth and GC of the heap. It does not show any out-of-memory condition. It may simply be that you need to increase the memory you allocate, especially the permanent generation (you should probably read up on this). Unless you can point to an actual memory leak from a heap dump, I'd like to close this. > Repeated sc.close() in PySpark causes JVM memory leak > ----------------------------------------------------- > > Key: SPARK-9539 > URL: https://issues.apache.org/jira/browse/SPARK-9539 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 1.4.1 > Reporter: Andrey Zimovnov > Priority: Minor > Attachments: Screenshot at авг. 02 19-10-53.png > > > Example code in Python: > {code:python} > for i in range(20): > print i > conf = SparkConf().setAppName("test") > sc = SparkContext(conf=conf) > hivec = HiveContext(sc) > hivec.sql("select id from details_info limit 1").show() > sc.stop() > del hivec > del sc > {code} > Jstat output: > {noformat} > S0C S1C S0U S1U EC EU OC OU PC > PU YGC YGCT FGC FGCT GCT > 196608,0 196608,0 97566,2 0,0 1179648,0 542150,0 3145728,0 120,0 > 154112,0 153613,2 4 0,434 0 0,000 0,434 > 196608,0 196608,0 97566,2 0,0 1179648,0 679041,7 3145728,0 120,0 > 164352,0 164183,3 4 0,434 0 0,000 0,434 > 196608,0 196608,0 97566,2 0,0 1179648,0 907928,4 3145728,0 120,0 > 164352,0 164200,3 4 0,434 0 0,000 0,434 > 196608,0 196608,0 97566,2 0,0 1179648,0 912132,7 3145728,0 120,0 > 164352,0 164200,5 4 0,434 0 0,000 0,434 > 196608,0 196608,0 97566,2 0,0 1179648,0 913741,5 3145728,0 120,0 > 164352,0 164200,8 4 0,434 0 0,000 0,434 > 196608,0 196608,0 97566,2 0,0 1179648,0 929458,6 3145728,0 120,0 > 164352,0 164206,0 4 0,434 0 0,000 0,434 > 196608,0 196608,0 97566,2 0,0 1179648,0 1003138,1 3145728,0 120,0 > 168960,0 168646,0 4 0,434 0 0,000 0,434 > 131584,0 196608,0 0,0 109725,6 1179648,0 0,0 3145728,0 128,0 > 175104,0 174802,1 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 152654,9 3145728,0 128,0 > 175104,0 174803,3 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 158586,1 3145728,0 128,0 > 175104,0 174803,3 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 160659,8 3145728,0 128,0 > 175104,0 174805,7 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 181935,2 3145728,0 128,0 > 175104,0 174819,7 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 283389,1 3145728,0 128,0 > 185856,0 185371,0 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 342596,4 3145728,0 128,0 > 185856,0 185379,3 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 547634,7 3145728,0 128,0 > 185856,0 185385,8 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 555930,9 3145728,0 128,0 > 185856,0 185385,8 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 557888,6 3145728,0 128,0 > 185856,0 185386,0 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 573907,5 3145728,0 128,0 > 185856,0 185397,5 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 637955,0 3145728,0 128,0 > 189952,0 189533,1 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 895866,1 3145728,0 128,0 > 196096,0 195968,5 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 948046,5 3145728,0 128,0 > 196096,0 195969,4 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 952427,2 3145728,0 128,0 > 196096,0 195969,4 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 957977,5 3145728,0 128,0 > 196096,0 195973,4 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 977811,1 3145728,0 128,0 > 196096,0 195977,7 5 0,591 0 0,000 0,591 > 131584,0 196608,0 0,0 109725,6 1179648,0 1118722,0 3145728,0 128,0 > 206848,0 206539,0 5 0,591 0 0,000 0,591 > 131584,0 144384,0 118692,5 0,0 1284096,0 183470,8 3145728,0 136,0 > 206848,0 206543,4 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 189718,5 3145728,0 136,0 > 206848,0 206543,4 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 192165,0 3145728,0 136,0 > 206848,0 206543,4 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 199848,4 3145728,0 136,0 > 206848,0 206546,9 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 219687,6 3145728,0 136,0 > 206848,0 206552,2 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 358272,4 3145728,0 136,0 > 217600,0 217100,4 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 573543,6 3145728,0 136,0 > 217600,0 217109,4 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 593569,9 3145728,0 136,0 > 217600,0 217109,6 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 593576,3 3145728,0 136,0 > 217600,0 217109,6 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 608832,1 3145728,0 136,0 > 217600,0 217114,6 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 649072,6 3145728,0 136,0 > 218624,0 218234,7 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 774293,0 3145728,0 136,0 > 227840,0 227734,6 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 971194,9 3145728,0 136,0 > 227840,0 227739,2 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 992291,6 3145728,0 136,0 > 227840,0 227739,2 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 992297,7 3145728,0 136,0 > 227840,0 227739,2 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 1002356,2 3145728,0 136,0 > 227840,0 227745,1 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 1020944,8 3145728,0 136,0 > 227840,0 227754,0 6 0,773 0 0,000 0,773 > 131584,0 144384,0 118692,5 0,0 1284096,0 1169159,0 3145728,0 136,0 > 238592,0 238332,9 6 0,773 0 0,000 0,773 > 147456,0 109056,0 0,0 108867,6 1284096,0 110659,1 3145728,0 19167,5 > 238592,0 238479,1 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 116954,2 3145728,0 19167,5 > 238592,0 238491,0 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 116960,6 3145728,0 19167,5 > 238592,0 238491,0 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 132244,5 3145728,0 19167,5 > 238592,0 238491,9 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 218816,2 3145728,0 19167,5 > 245760,0 244549,2 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 467076,0 3145728,0 19167,5 > 249344,0 249189,3 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 508023,8 3145728,0 19167,5 > 249344,0 249189,8 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 511485,1 3145728,0 19167,5 > 249344,0 249191,3 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 513094,4 3145728,0 19167,5 > 249344,0 249191,9 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 527507,0 3145728,0 19167,5 > 249344,0 249197,0 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 636565,3 3145728,0 19167,5 > 260096,0 259705,2 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 819041,0 3145728,0 19167,5 > 260096,0 259749,7 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 917166,3 3145728,0 19167,5 > 260096,0 259749,7 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 919595,7 3145728,0 19167,5 > 260096,0 259749,7 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 925158,0 3145728,0 19167,5 > 260096,0 259750,1 7 0,942 0 0,000 0,942 > 147456,0 109056,0 0,0 108867,6 1284096,0 946351,0 3145728,0 19167,5 > 260096,0 259751,1 7 0,942 0 0,000 0,942 > 151552,0 1024,0 0,0 924,7 1284608,0 0,0 3145728,0 119860,5 > 262144,0 262143,9 9 1,038 2 2,021 3,059 > 148480,0 153088,0 0,0 64,0 1262080,0 0,0 3145728,0 84562,9 > 262144,0 262143,9 17 1,167 10 6,988 8,155 > 99840,0 93696,0 160,0 0,0 1379328,0 0,0 3145728,0 84035,5 > 262144,0 262143,6 26 1,296 19 12,022 13,318 > 50688,0 54272,0 0,0 224,0 1460224,0 0,0 3145728,0 83958,1 > 262144,0 262143,8 35 1,474 28 16,642 18,116 > 23552,0 25088,0 0,0 96,0 1520640,0 0,0 3145728,0 84007,6 > 262144,0 262144,0 45 1,638 38 21,492 23,130 > {noformat} > It looks like it hits -XX:MaxPermSize=256m for the driver and dies with OOM: > Exception: ("You must build Spark with Hive. Export 'SPARK_HIVE=true' and run > build/sbt assembly", Py4JJavaError(u'An error occurred while calling > None.org.apache.spark.sql.hive.HiveContext.\n', JavaObject id=o753)) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org