----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59868/ -----------------------------------------------------------
(Updated June 7, 2017, 5:07 p.m.) Review request for Aurora, David McLaughlin, Jordan Ly, Kai Huang, Santhosh Kumar Shanmugham, and Stephan Erb. Repository: aurora Description ------- # Allow disk monitoring to be disabled in Observer https://docs.google.com/document/d/1-1eYenw9wgsWXyyOWn32l0oJBBXpzIN5V20vi0AZpi8/edit?usp=sharing This is part(2) of the above proposal. --- ## recreating the problem I tested it with a task that creates a large number of nested directories (300,000 to be exact). When disk monitoring is enabled the Observer logs show: > D0607 00:09:08.082673 24209 disk.py:44] DiskCollectorThread: finished > collection of > /var/lib/mesos/slaves/6ca6dce6-6906-41a2-a9ba-215c703ac349-S0/frameworks/6ca6dce6-6906-41a2-a9ba-215c703ac349-0000/executors/thermos-www-data-devel-hello-0-21163fcd-a2c7-46c3-ab48-cc83c7396394/runs/12917d81-500b-49c0-a1d2-5aeee8b2147b/sandbox > in __26884.2ms__. Note that at this point all the di Also running `top`. ``` PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24209 root 20 0 1988376 32148 4100 S 69.0 1.0 1:38.68 python2.7 /home/vagrant/aurora/dist/thermos_observer.pex --ip=192.168.33.7 --port=1338 --log_to_disk=INFO --log_to_stderr=google:DEBUG 16909 root 20 0 1142712 43596 8144 S 68.4 1.4 0:47.75 python2.7 /var/lib/mesos/slaves/6ca6dce6-6906-41a2-a9ba-215c703ac349-S0/frameworks/6ca6dce6-6906-41a2-a9ba-215c703ac349-0000/executors/thermos-www-data-devel-hello-0+ ``` I saw observer cpu usage going up to 89%. Diffs (updated) ----- src/main/python/apache/aurora/tools/thermos_observer.py 0318f990ac003c0b8925b7eb7359431cdee34f05 src/main/python/apache/thermos/monitoring/disk.py 52c5d74fd70b5942ea3ef5101ba3f27bfc98fc21 src/main/python/apache/thermos/observer/task_observer.py 4bb5d239e81fe4659397f899760c0e8853e93786 Diff: https://reviews.apache.org/r/59868/diff/2/ Changes: https://reviews.apache.org/r/59868/diff/1-2/ Testing ------- ``` # rmotamedi@tw-mbp-rmotamedi:~/oss/aurora on git:disable-observer-du [17:38:41] ? ./build-support/jenkins/build.sh + date Tue Jun 6 17:39:26 PDT 2017 + ./gradlew -Pq clean build Starting a Gradle Daemon (subsequent builds will be faster) :buildSrc:compileJava UP-TO-DATE :buildSrc:compileGroovy UP-TO-DATE :buildSrc:processResources UP-TO-DATE ... .. . :commons-args:testClasses UP-TO-DATE :commons-args:test UP-TO-DATE :commons-args:check UP-TO-DATE :commons-args:build BUILD SUCCESSFUL ``` I also added `--disable-disk-monitor` to `examples/vagrant/upstart/aurora-thermos-observer.conf` and traced the change to the observer UI. The attached screen shot shows that disk `-0.0GB`. File Attachments ---------------- Screen Shot 2017-06-06 at 5.35.36 PM.png https://reviews.apache.org/media/uploaded/files/2017/06/07/d3b77673-f9f1-45c8-a52c-b2787322d769__Screen_Shot_2017-06-06_at_5.35.36_PM.png Thanks, Reza Motamedi
