[
https://issues.apache.org/jira/browse/SPARK-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999601#comment-13999601
]
Patrick Wendell edited comment on SPARK-1154 at 5/16/14 5:00 AM:
-----------------------------------------------------------------
[~mkim] yes you are correct - this is broken. Checkout SPARK-1860. Are you
interested in fixing this?
was (Author: pwendell):
[~mkim] yes you are correct - this is broken. Checkout SPARK-1860.
> Spark fills up disk with app-* folders
> --------------------------------------
>
> Key: SPARK-1154
> URL: https://issues.apache.org/jira/browse/SPARK-1154
> Project: Spark
> Issue Type: Improvement
> Components: Deploy
> Reporter: Evan Chan
> Assignee: Evan Chan
> Priority: Critical
> Labels: starter
> Fix For: 1.0.0
>
>
> Current version of Spark fills up the disk with many app-* folders:
> $ ls /var/lib/spark
> app-20140210022347-0597 app-20140212173327-0627 app-20140218154110-0657
> app-20140225232537-0017 app-20140225233548-0047
> app-20140210022407-0598 app-20140212173347-0628 app-20140218154130-0658
> app-20140225232551-0018 app-20140225233556-0048
> app-20140210022427-0599 app-20140212173754-0629 app-20140218164232-0659
> app-20140225232611-0019 app-20140225233603-0049
> app-20140210022447-0600 app-20140212182235-0630 app-20140218165133-0660
> app-20140225232802-0020 app-20140225233610-0050
> app-20140210022508-0601 app-20140212182256-0631 app-20140218165148-0661
> app-20140225232822-0021 app-20140225233617-0051
> app-20140210022528-0602 app-20140213000014-0632 app-20140218165225-0662
> app-20140225232940-0022 app-20140225233624-0052
> app-20140211024356-0603 app-20140213002026-0633 app-20140218165249-0663
> app-20140225233002-0023 app-20140225233631-0053
> app-20140211024417-0604 app-20140213154948-0634 app-20140218172030-0664
> app-20140225233056-0024 app-20140225233725-0054
> app-20140211024437-0605 app-20140213171810-0635 app-20140218193853-0665
> app-20140225233108-0025 app-20140225233731-0055
> app-20140211024457-0606 app-20140213193637-0636 app-20140218194442-0666
> app-20140225233124-0026 app-20140225233733-0056
> app-20140211024517-0607 app-20140214011513-0637 app-20140218194746-0667
> app-20140225233133-0027 app-20140225233734-0057
> app-20140211024538-0608 app-20140214012151-0638 app-20140218194822-0668
> app-20140225233147-0028 app-20140225233749-0058
> app-20140211193443-0609 app-20140214013134-0639 app-20140218212317-0669
> app-20140225233208-0029 app-20140225233759-0059
> app-20140211195210-0610 app-20140214013332-0640 app-20140225180142-0000
> app-20140225233215-0030 app-20140225233809-0060
> app-20140211213935-0611 app-20140214013642-0641 app-20140225180411-0001
> app-20140225233224-0031 app-20140225233828-0061
> app-20140211214227-0612 app-20140214014246-0642 app-20140225180431-0002
> app-20140225233232-0032 app-20140225234719-0062
> app-20140211215317-0613 app-20140214014607-0643 app-20140225180452-0003
> app-20140225233239-0033 app-20140226032845-0063
> app-20140211224601-0614 app-20140214184943-0644 app-20140225180512-0004
> app-20140225233320-0034 app-20140226033004-0064
> app-20140212022206-0615 app-20140214185118-0645 app-20140225180533-0005
> app-20140225233328-0035 app-20140226033119-0065
> app-20140212022226-0616 app-20140214185851-0646 app-20140225180553-0006
> app-20140225233354-0036 app-20140226033334-0066
> app-20140212022246-0617 app-20140214222856-0647 app-20140225181115-0007
> app-20140225233402-0037 app-20140226033354-0067
> app-20140212043704-0618 app-20140214231312-0648 app-20140225181244-0008
> app-20140225233409-0038 app-20140226033538-0068
> app-20140212043724-0619 app-20140214231434-0649 app-20140225182051-0009
> app-20140225233416-0039 app-20140226033826-0069
> app-20140212043745-0620 app-20140214231542-0650 app-20140225183009-0010
> app-20140225233426-0040 app-20140226034002-0070
> app-20140212044016-0621 app-20140214231616-0651 app-20140225184133-0011
> app-20140225233432-0041 app-20140226034053-0071
> app-20140212044203-0622 app-20140214233016-0652 app-20140225184318-0012
> app-20140225233439-0042 app-20140226034234-0072
> app-20140212044224-0623 app-20140214233037-0653 app-20140225184709-0013
> app-20140225233447-0043 app-20140226034426-0073
> app-20140212045034-0624 app-20140218153242-0654 app-20140225184844-0014
> app-20140225233526-0044 app-20140226034447-0074
> app-20140212045119-0625 app-20140218153341-0655 app-20140225190051-0015
> app-20140225233534-0045
> app-20140212173310-0626 app-20140218153442-0656 app-20140225232516-0016
> app-20140225233540-0046
> This problem is particularly bad if you have a whole bunch of fast jobs.
> Also what makes the problem worse is that any jars for jobs is downloaded
> into the app-* folder, so that fills up the disk particularly fast.
> I would like to propose two things:
> 1) Spark should have a cleanup thread (or actor) which periodically removes
> old app-* folders; This should not be the responsibility of people
> deploying Spark.
> 2) The download of jars should not go to each app-* folder. This wastes a
> huge amount of space because most jobs use the same jars. Maybe I can open a
> separate ticket for this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
