-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59868/#review177151
-----------------------------------------------------------
Master (73234d0) is red with this patch.
./build-support/jenkins/build.sh
Running setup.py bdist_wheel for pycparser: finished with status 'done'
Stored in directory:
/home/jenkins/jenkins-slave/workspace/AuroraBot/.home/.cache/pip/wheels/a8/0b/41/dc95621f9d3a0da7bc191b8a71f0e8182ffd3cc5f33ac55005
Running setup.py bdist_wheel for twitter.common.options: started
Running setup.py bdist_wheel for twitter.common.options: finished with status
'done'
Stored in directory:
/home/jenkins/jenkins-slave/workspace/AuroraBot/.home/.cache/pip/wheels/17/41/80/c4811d8c1c7ca7007e520c3399872fc340f45c3a26a6a23e6a
Successfully built pantsbuild.pants twitter.common.collections setproctitle
ansicolors pathspec scandir twitter.common.dirutil psutil pystache docutils
Markdown Pygments twitter.common.confluence coverage lmdb pywatchman
twitter.common.lang twitter.common.log pycparser twitter.common.options
Installing collected packages: twitter.common.lang, twitter.common.collections,
setproctitle, setuptools, six, ansicolors, pyparsing, packaging, pathspec,
scandir, twitter.common.dirutil, psutil, requests, pystache, pex, docutils,
Markdown, Pygments, twitter.common.options, twitter.common.log,
twitter.common.confluence, monotonic, fasteners, coverage, pycparser, cffi,
lmdb, pywatchman, futures, pantsbuild.pants
Found existing installation: setuptools 21.2.1
Uninstalling setuptools-21.2.1:
Successfully uninstalled setuptools-21.2.1
Successfully installed Markdown-2.1.1 Pygments-1.4 ansicolors-1.0.2 cffi-1.7.0
coverage-3.7.1 docutils-0.12 fasteners-0.14.1 futures-3.0.5 lmdb-0.89
monotonic-1.3 packaging-16.7 pantsbuild.pants-1.3.0.dev3 pathspec-0.3.4
pex-1.1.16 psutil-4.3.0 pycparser-2.17 pyparsing-2.2.0 pystache-0.5.3
pywatchman-1.3.0 requests-2.5.3 scandir-1.2 setproctitle-1.1.10
setuptools-30.0.0 six-1.10.0 twitter.common.collections-0.3.9
twitter.common.confluence-0.3.9 twitter.common.dirutil-0.3.9
twitter.common.lang-0.3.9 twitter.common.log-0.3.9 twitter.common.options-0.3.9
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
05:27:03 00:00 [main]
(To run a reporting server: ./pants server)
05:27:03 00:00 [setup]
05:27:03 00:00 [parse][32m
Executing tasks in goals: compile[0m
05:27:03 00:00 [compile]
05:27:03 00:00 [compile-prep-command]
05:27:03 00:00 [prep_command]
05:27:06 00:03 [compile]
05:27:06 00:03 [python-eval]
05:27:06 00:03 [pythonstyle]
05:27:06 00:03 [cache] [32m
No cached artifacts for 42 targets.[0m[32m
Invalidated 42 targets.[0m
T302:ERROR src/main/python/apache/aurora/tools/thermos_observer.py:087
Expected 2 blank lines, found 1
|def sleep_forever():
E302:ERROR
PythonFile(src/main/python/apache/aurora/tools/thermos_observer.py):087
expected 2 blank lines, found 1
|def sleep_forever():
FAILURE: 2 Python Style issues found. For import order related issues, please
try `./pants fmt.isort <targets>`
05:27:29 00:26 [complete][31m
FAILURE[0m
I will refresh this build result if you post a review containing "@ReviewBot
retry"
- Aurora ReviewBot
On June 7, 2017, 7:14 a.m., Reza Motamedi wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59868/
> -----------------------------------------------------------
>
> (Updated June 7, 2017, 7:14 a.m.)
>
>
> Review request for Aurora, David McLaughlin, Jordan Ly, Kai Huang, Santhosh
> Kumar Shanmugham, and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> # Allow disk monitoring to be disabled in Observer
>
> https://docs.google.com/document/d/1-1eYenw9wgsWXyyOWn32l0oJBBXpzIN5V20vi0AZpi8/edit?usp=sharing
>
> This is part(2) of the above proposal.
>
> ---
>
> ## recreating the problem
>
> I tested it with a task that creates a large number of nested directories
> (300,000 to be exact). When disk monitoring is enabled the Observer logs show:
> > D0607 00:09:08.082673 24209 disk.py:44] DiskCollectorThread: finished
> > collection of
> > /var/lib/mesos/slaves/6ca6dce6-6906-41a2-a9ba-215c703ac349-S0/frameworks/6ca6dce6-6906-41a2-a9ba-215c703ac349-0000/executors/thermos-www-data-devel-hello-0-21163fcd-a2c7-46c3-ab48-cc83c7396394/runs/12917d81-500b-49c0-a1d2-5aeee8b2147b/sandbox
> > in __26884.2ms__.
>
> Note that at this point all the di
> Also running `top`.
>
> ```
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 24209 root 20 0 1988376 32148 4100 S 69.0 1.0 1:38.68 python2.7
> /home/vagrant/aurora/dist/thermos_observer.pex --ip=192.168.33.7 --port=1338
> --log_to_disk=INFO --log_to_stderr=google:DEBUG
> 16909 root 20 0 1142712 43596 8144 S 68.4 1.4 0:47.75 python2.7
> /var/lib/mesos/slaves/6ca6dce6-6906-41a2-a9ba-215c703ac349-S0/frameworks/6ca6dce6-6906-41a2-a9ba-215c703ac349-0000/executors/thermos-www-data-devel-hello-0+
> ```
>
> I saw observer cpu usage going up to 89%.
>
>
> Diffs
> -----
>
> src/main/python/apache/aurora/tools/thermos_observer.py
> 0318f990ac003c0b8925b7eb7359431cdee34f05
> src/main/python/apache/thermos/monitoring/disk.py
> 52c5d74fd70b5942ea3ef5101ba3f27bfc98fc21
> src/main/python/apache/thermos/observer/task_observer.py
> 4bb5d239e81fe4659397f899760c0e8853e93786
>
>
> Diff: https://reviews.apache.org/r/59868/diff/1/
>
>
> Testing
> -------
>
> ```
> # rmotamedi@tw-mbp-rmotamedi:~/oss/aurora on git:disable-observer-du
> [17:38:41]
> ? ./build-support/jenkins/build.sh
> + date
> Tue Jun 6 17:39:26 PDT 2017
> + ./gradlew -Pq clean build
> Starting a Gradle Daemon (subsequent builds will be faster)
> :buildSrc:compileJava UP-TO-DATE
> :buildSrc:compileGroovy UP-TO-DATE
> :buildSrc:processResources UP-TO-DATE
> ...
> ..
> .
> :commons-args:testClasses UP-TO-DATE
> :commons-args:test UP-TO-DATE
> :commons-args:check UP-TO-DATE
> :commons-args:build
>
> BUILD SUCCESSFUL
> ```
>
> I also added `--disable-disk-monitor` to
> `examples/vagrant/upstart/aurora-thermos-observer.conf` and traced the change
> to the observer UI. The attached screen shot shows that disk `-0.0GB`.
>
>
> File Attachments
> ----------------
>
> Screen Shot 2017-06-06 at 5.35.36 PM.png
>
> https://reviews.apache.org/media/uploaded/files/2017/06/07/d3b77673-f9f1-45c8-a52c-b2787322d769__Screen_Shot_2017-06-06_at_5.35.36_PM.png
>
>
> Thanks,
>
> Reza Motamedi
>
>
