[
https://issues.apache.org/jira/browse/SPARK-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225638#comment-14225638
]
Andrew Or commented on SPARK-3546:
----------------------------------
Hey [~sarutak] do we want this for branch-1.1? If so we should re-open this.
> InputStream of ManagedBuffer is not closed and causes running out of file
> descriptor
> ------------------------------------------------------------------------------------
>
> Key: SPARK-3546
> URL: https://issues.apache.org/jira/browse/SPARK-3546
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Kousuke Saruta
> Assignee: Kousuke Saruta
> Priority: Critical
> Fix For: 1.2.0
>
>
> If application has lots of shuffle blocks, resource leak (running out of file
> descriptor) is occurred.
> Following text is file descriptors of an Executor which has lots of blocks
> {code}
> ・・・
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9980 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/13/shuffle_0_340_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9981 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/37/shuffle_0_355_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9982 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/10/shuffle_0_370_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9983 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/0c/shuffle_0_385_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9984 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/0e/shuffle_0_390_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9985 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/1d/shuffle_0_405_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9986 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/36/shuffle_0_420_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9987 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/1b/shuffle_0_425_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9988 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/0b/shuffle_0_430_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9989 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/0d/shuffle_0_450_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:11 2014 999 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/28/shuffle_1_630_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9990 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/29/shuffle_0_465_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9991 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/14/shuffle_0_495_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9992 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/3c/shuffle_0_525_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9993 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/2a/shuffle_0_530_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9994 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/05/shuffle_0_535_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9995 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/15/shuffle_0_540_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9996 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/2c/shuffle_0_550_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9997 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/13/shuffle_0_560_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9998 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/12/shuffle_0_570_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:27 2014 9999 ->
> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/2f/shuffle_0_580_0.data
> {code}
> This is caused by not closing InputStream generated by ManagedBuffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
