Dmytro Sen created AMBARI-9771:
----------------------------------
Summary: AMS: ResultSpooler spill files are left behind in /tmp
folder
Key: AMBARI-9771
URL: https://issues.apache.org/jira/browse/AMBARI-9771
Project: Ambari
Issue Type: Bug
Components: ambari-metrics
Affects Versions: 2.0.0
Reporter: Dmytro Sen
Assignee: Dmytro Sen
Priority: Critical
Fix For: 2.0.0
I've seen behavior, described in PHOENIX-1395, on Ambari cluster (200 nodes)
After 3 days running, /tmp folder is full of ResultSpool files
{noformat}
[root@perf-200-20feb-b-ams-1 ~]# ls -la /tmp/
total 4238768
drwxrwxrwt. 6 root root 4096 Feb 23 10:13 .
dr-xr-xr-x. 24 root root 4096 Feb 20 14:55 ..
drwxr-xr-x. 2 hdfs root 4096 Feb 20 16:46 hadoop-hdfs
drwxr-xr-x. 2 ams hadoop 4096 Feb 20 18:19 hsperfdata_ams
drwxrwxrwt. 2 root root 4096 Feb 20 14:16 .ICE-unix
drwxr-xr-x. 5 ams hadoop 4096 Feb 20 18:20
Jetty_0_0_0_0_6188_applicationhistory____g7149g
-rw-r--r--. 1 root root 617 Feb 20 14:16 keyfile.1115.pub
-rwxrwxrwx. 1 root root 2557032 Feb 20 17:59 libyjpagent.so
-rw-r--r--. 1 ams hadoop 16676205 Feb 22 11:36
ResultSpooler1083719173569401088.bin
-rw-r--r--. 1 ams hadoop 71227095 Feb 22 02:10
ResultSpooler1095642585411691664.bin
-rw-r--r--. 1 ams hadoop 48685977 Feb 22 11:32
ResultSpooler1119370459591752039.bin
-rw-r--r--. 1 ams hadoop 64366788 Feb 21 02:24
ResultSpooler1249566735083013570.bin
-rw-r--r--. 1 ams hadoop 102519737 Feb 21 15:15
ResultSpooler1434399174629975447.bin
-rw-r--r--. 1 ams hadoop 62280357 Feb 21 07:04
ResultSpooler1523399749056056116.bin
-rw-r--r--. 1 ams hadoop 99096145 Feb 21 23:28
ResultSpooler1591731371355221216.bin
-rw-r--r--. 1 ams hadoop 33274946 Feb 22 07:42
ResultSpooler1763163145889847809.bin
-rw-r--r--. 1 ams hadoop 29906801 Feb 22 06:45
ResultSpooler1924668033838582502.bin
-rw-r--r--. 1 ams hadoop 79817760 Feb 20 18:19
ResultSpooler2061194340898298157.bin
-rw-r--r--. 1 ams hadoop 39832796 Feb 22 11:32
ResultSpooler2084948115978783352.bin
-rw-r--r--. 1 ams hadoop 16340565 Feb 22 11:07
ResultSpooler2151011123128612740.bin
-rw-r--r--. 1 ams hadoop 71125739 Feb 21 12:44
ResultSpooler227400912270327467.bin
-rw-r--r--. 1 ams hadoop 47016101 Feb 21 15:15
ResultSpooler2397265359677018278.bin
-rw-r--r--. 1 ams hadoop 96264436 Feb 22 02:10
ResultSpooler2614183649855534427.bin
-rw-r--r--. 1 ams hadoop 69682164 Feb 20 18:19
ResultSpooler2614354335568461108.bin
-rw-r--r--. 1 ams hadoop 16215764 Feb 22 06:45
ResultSpooler3246758863501341303.bin
-rw-r--r--. 1 ams hadoop 148128967 Feb 20 20:50
ResultSpooler3488123938708977657.bin
-rw-r--r--. 1 ams hadoop 25723333 Feb 22 06:46
ResultSpooler3579970151989744696.bin
-rw-r--r--. 1 ams hadoop 21823041 Feb 22 07:42
ResultSpooler3777629950823001041.bin
-rw-r--r--. 1 ams hadoop 72062056 Feb 20 18:04
ResultSpooler3852071445687640038.bin
-rw-r--r--. 1 ams hadoop 81368534 Feb 21 01:22
ResultSpooler3875643246636160232.bin
-rw-r--r--. 1 ams hadoop 30168849 Feb 22 10:25
ResultSpooler3965157245482339436.bin
-rw-r--r--. 1 ams hadoop 196130729 Feb 20 18:08
ResultSpooler4121123109947014705.bin
-rw-r--r--. 1 ams hadoop 23298601 Feb 22 06:45
ResultSpooler4193988685099849898.bin
-rw-r--r--. 1 ams hadoop 24327567 Feb 22 11:08
ResultSpooler435113448003575788.bin
-rw-r--r--. 1 ams hadoop 98370825 Feb 21 20:57
ResultSpooler4731594258931869098.bin
-rw-r--r--. 1 ams hadoop 81147072 Feb 22 05:17
ResultSpooler4839918557747068904.bin
-rw-r--r--. 1 ams hadoop 16321500 Feb 22 11:40
ResultSpooler4948059449385475969.bin
-rw-r--r--. 1 ams hadoop 26490333 Feb 22 11:32
ResultSpooler5183367322708173382.bin
-rw-r--r--. 1 ams hadoop 41887967 Feb 22 02:09
ResultSpooler5363100158650913466.bin
-rw-r--r--. 1 ams hadoop 61564579 Feb 22 03:22
ResultSpooler5447269850518103716.bin
-rw-r--r--. 1 ams hadoop 102368545 Feb 21 05:23
ResultSpooler5708767266121623071.bin
-rw-r--r--. 1 ams hadoop 27115177 Feb 21 12:24
ResultSpooler5742673986917724242.bin
-rw-r--r--. 1 ams hadoop 136561367 Feb 22 02:48
ResultSpooler5764784857800306062.bin
-rw-r--r--. 1 ams hadoop 72956934 Feb 21 00:05
ResultSpooler5802156014065836906.bin
-rw-r--r--. 1 ams hadoop 158032579 Feb 20 20:20
ResultSpooler5957862203905947616.bin
-rw-r--r--. 1 ams hadoop 163713067 Feb 21 15:17
ResultSpooler6028348699797622192.bin
-rw-r--r--. 1 ams hadoop 74337617 Feb 21 15:17
ResultSpooler6513985637309952930.bin
-rw-r--r--. 1 ams hadoop 151691821 Feb 22 05:17
ResultSpooler6722852862052886883.bin
-rw-r--r--. 1 ams hadoop 158153349 Feb 22 02:12
ResultSpooler6785318486215835763.bin
-rw-r--r--. 1 ams hadoop 63085961 Feb 21 22:22
ResultSpooler7037247582202398147.bin
-rw-r--r--. 1 ams hadoop 25750043 Feb 22 11:07
ResultSpooler7161010036991333211.bin
-rw-r--r--. 1 ams hadoop 23637237 Feb 22 11:32
ResultSpooler7265305658142312639.bin
-rw-r--r--. 1 ams hadoop 23514267 Feb 22 10:25
ResultSpooler7273301530051464668.bin
-rw-r--r--. 1 ams hadoop 23725265 Feb 22 07:41
ResultSpooler7435219196600981746.bin
-rw-r--r--. 1 ams hadoop 74368269 Feb 21 07:04
ResultSpooler7530898046096394659.bin
-rw-r--r--. 1 ams hadoop 20624014 Feb 20 18:00
ResultSpooler7621079742935581555.bin
-rw-r--r--. 1 ams hadoop 16321827 Feb 22 07:41
ResultSpooler7731471884268988990.bin
-rw-r--r--. 1 ams hadoop 134992041 Feb 21 23:28
ResultSpooler7749761859495380220.bin
-rw-r--r--. 1 ams hadoop 16246582 Feb 22 11:32
ResultSpooler7883369468940765812.bin
-rw-r--r--. 1 ams hadoop 38676810 Feb 22 11:07
ResultSpooler8060777733113377059.bin
-rw-r--r--. 1 ams hadoop 66444927 Feb 21 20:22
ResultSpooler8171687820667276916.bin
-rw-r--r--. 1 ams hadoop 101361125 Feb 21 22:00
ResultSpooler8235372529929072797.bin
-rw-r--r--. 1 ams hadoop 98019115 Feb 20 21:47
ResultSpooler8258396922291011919.bin
-rw-r--r--. 1 ams hadoop 98958044 Feb 21 14:34
ResultSpooler841310175906717024.bin
-rw-r--r--. 1 ams hadoop 37453544 Feb 21 07:02
ResultSpooler8476786944095537854.bin
-rw-r--r--. 1 ams hadoop 40917633 Feb 22 10:26
ResultSpooler86283493320653508.bin
-rw-r--r--. 1 ams hadoop 16339554 Feb 22 10:25
ResultSpooler8650910217102203243.bin
-rw-r--r--. 1 ams hadoop 33496406 Feb 21 23:03
ResultSpooler8840476939367711195.bin
-rw-r--r--. 1 ams hadoop 34168131 Feb 22 10:26
ResultSpooler8866025986224263370.bin
-rw-r--r--. 1 ams hadoop 145475909 Feb 20 23:19
ResultSpooler908545909889857121.bin
-rw-r--r--. 1 ams hadoop 145899169 Feb 20 21:49
ResultSpooler9088480868111255890.bin
-rw-r--r--. 1 ams hadoop 100219001 Feb 20 20:19
ResultSpooler9184998486224339551.bin
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)