Re: Review Request 58609: Switch Thermos runner to simple disk log layout

[Stephan Erb]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58609/#review172671
---



@ReviewBot retry

- Stephan Erb


On April 21, 2017, 11:35 a.m., Stephan Erb wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58609/
> ---
> 
> (Updated April 21, 2017, 11:35 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Switch Thermos runner to simple disk log layout.
> 
> This collapses the individual files per loglevel. We will therefore have just 
> one log file for executor and one for the runner.
> 
> -- 
> 
> Not to reviwers: I am unsure why the executor log file defaults to __main__ 
> rather than the name of the binary. I decided not to dig deeper here, as it 
> is not my main problem during operations.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/executor/thermos_task_runner.py 
> 8f88af4c24ddc603fa12587741af56a6c711e420 
> 
> 
> Diff: https://reviews.apache.org/r/58609/diff/1/
> 
> 
> Testing
> ---
> 
> ./pants test.pytest src/test/python::
> 
> 
> File Attachments
> 
> 
> after.png
>   
> https://reviews.apache.org/media/uploaded/files/2017/04/21/8816f094-b473-45bd-8d9a-fd2f3fad7dfe__after.png
> before.png
>   
> https://reviews.apache.org/media/uploaded/files/2017/04/21/f10b3500-a66b-4850-a571-f964b97b7b40__before.png
> 
> 
> Thanks,
> 
> Stephan Erb
> 
>

Re: Review Request 58609: Switch Thermos runner to simple disk log layout

[Joshua Cohen]

> On April 21, 2017, 2:37 p.m., Joshua Cohen wrote:
> > src/main/python/apache/aurora/executor/thermos_task_runner.py
> > Line 257 (original), 257 (patched)
> > 
> >
> > I think a better place to do this might be in 
> > [thermos_runner.py](https://github.com/apache/aurora/blob/master/src/main/python/apache/thermos/runner/thermos_runner.py).
> > 
> > Instead of passing this as a command line arg, we could just:
> > 
> > from twitter.common.log.options import LogOptions
> > 
> > LogOptions.set_simple(True)
> > 
> > (We already do this in our internal fork, no idea why it's done there 
> > instead of the main project)
> 
> Stephan Erb wrote:
> There are some further options a few lines above:
> 
> ```
> params = dict(log_dir=LogOptions.log_dir(),
>   log_to_disk='DEBUG',
> ```
> 
> 
> Should I move those as well?
> 
> Stephan Erb wrote:
> Ah thinking about it, we probably still need the log_dir, to transfer the 
> log location from the executor to the runner.

Yeah, I'm fine with moving `log_to_disk` if you want though.


- Joshua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58609/#review172657
---


On April 21, 2017, 9:35 a.m., Stephan Erb wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58609/
> ---
> 
> (Updated April 21, 2017, 9:35 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Switch Thermos runner to simple disk log layout.
> 
> This collapses the individual files per loglevel. We will therefore have just 
> one log file for executor and one for the runner.
> 
> -- 
> 
> Not to reviwers: I am unsure why the executor log file defaults to __main__ 
> rather than the name of the binary. I decided not to dig deeper here, as it 
> is not my main problem during operations.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/executor/thermos_task_runner.py 
> 8f88af4c24ddc603fa12587741af56a6c711e420 
> 
> 
> Diff: https://reviews.apache.org/r/58609/diff/1/
> 
> 
> Testing
> ---
> 
> ./pants test.pytest src/test/python::
> 
> 
> File Attachments
> 
> 
> after.png
>   
> https://reviews.apache.org/media/uploaded/files/2017/04/21/8816f094-b473-45bd-8d9a-fd2f3fad7dfe__after.png
> before.png
>   
> https://reviews.apache.org/media/uploaded/files/2017/04/21/f10b3500-a66b-4850-a571-f964b97b7b40__before.png
> 
> 
> Thanks,
> 
> Stephan Erb
> 
>

Re: Review Request 58609: Switch Thermos runner to simple disk log layout

[Stephan Erb]

> On April 21, 2017, 4:37 p.m., Joshua Cohen wrote:
> > src/main/python/apache/aurora/executor/thermos_task_runner.py
> > Line 257 (original), 257 (patched)
> > 
> >
> > I think a better place to do this might be in 
> > [thermos_runner.py](https://github.com/apache/aurora/blob/master/src/main/python/apache/thermos/runner/thermos_runner.py).
> > 
> > Instead of passing this as a command line arg, we could just:
> > 
> > from twitter.common.log.options import LogOptions
> > 
> > LogOptions.set_simple(True)
> > 
> > (We already do this in our internal fork, no idea why it's done there 
> > instead of the main project)
> 
> Stephan Erb wrote:
> There are some further options a few lines above:
> 
> ```
> params = dict(log_dir=LogOptions.log_dir(),
>   log_to_disk='DEBUG',
> ```
> 
> 
> Should I move those as well?

Ah thinking about it, we probably still need the log_dir, to transfer the log 
location from the executor to the runner.


- Stephan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58609/#review172657
---


On April 21, 2017, 11:35 a.m., Stephan Erb wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58609/
> ---
> 
> (Updated April 21, 2017, 11:35 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Switch Thermos runner to simple disk log layout.
> 
> This collapses the individual files per loglevel. We will therefore have just 
> one log file for executor and one for the runner.
> 
> -- 
> 
> Not to reviwers: I am unsure why the executor log file defaults to __main__ 
> rather than the name of the binary. I decided not to dig deeper here, as it 
> is not my main problem during operations.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/executor/thermos_task_runner.py 
> 8f88af4c24ddc603fa12587741af56a6c711e420 
> 
> 
> Diff: https://reviews.apache.org/r/58609/diff/1/
> 
> 
> Testing
> ---
> 
> ./pants test.pytest src/test/python::
> 
> 
> File Attachments
> 
> 
> after.png
>   
> https://reviews.apache.org/media/uploaded/files/2017/04/21/8816f094-b473-45bd-8d9a-fd2f3fad7dfe__after.png
> before.png
>   
> https://reviews.apache.org/media/uploaded/files/2017/04/21/f10b3500-a66b-4850-a571-f964b97b7b40__before.png
> 
> 
> Thanks,
> 
> Stephan Erb
> 
>

Re: Review Request 58609: Switch Thermos runner to simple disk log layout

[Stephan Erb]

> On April 21, 2017, 4:37 p.m., Joshua Cohen wrote:
> > src/main/python/apache/aurora/executor/thermos_task_runner.py
> > Line 257 (original), 257 (patched)
> > 
> >
> > I think a better place to do this might be in 
> > [thermos_runner.py](https://github.com/apache/aurora/blob/master/src/main/python/apache/thermos/runner/thermos_runner.py).
> > 
> > Instead of passing this as a command line arg, we could just:
> > 
> > from twitter.common.log.options import LogOptions
> > 
> > LogOptions.set_simple(True)
> > 
> > (We already do this in our internal fork, no idea why it's done there 
> > instead of the main project)

There are some further options a few lines above:

```
params = dict(log_dir=LogOptions.log_dir(),
  log_to_disk='DEBUG',
```


Should I move those as well?


- Stephan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58609/#review172657
---


On April 21, 2017, 11:35 a.m., Stephan Erb wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58609/
> ---
> 
> (Updated April 21, 2017, 11:35 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Switch Thermos runner to simple disk log layout.
> 
> This collapses the individual files per loglevel. We will therefore have just 
> one log file for executor and one for the runner.
> 
> -- 
> 
> Not to reviwers: I am unsure why the executor log file defaults to __main__ 
> rather than the name of the binary. I decided not to dig deeper here, as it 
> is not my main problem during operations.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/executor/thermos_task_runner.py 
> 8f88af4c24ddc603fa12587741af56a6c711e420 
> 
> 
> Diff: https://reviews.apache.org/r/58609/diff/1/
> 
> 
> Testing
> ---
> 
> ./pants test.pytest src/test/python::
> 
> 
> File Attachments
> 
> 
> after.png
>   
> https://reviews.apache.org/media/uploaded/files/2017/04/21/8816f094-b473-45bd-8d9a-fd2f3fad7dfe__after.png
> before.png
>   
> https://reviews.apache.org/media/uploaded/files/2017/04/21/f10b3500-a66b-4850-a571-f964b97b7b40__before.png
> 
> 
> Thanks,
> 
> Stephan Erb
> 
>

Re: Review Request 58609: Switch Thermos runner to simple disk log layout

[Joshua Cohen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58609/#review172657
---




src/main/python/apache/aurora/executor/thermos_task_runner.py
Line 257 (original), 257 (patched)


I think a better place to do this might be in 
[thermos_runner.py](https://github.com/apache/aurora/blob/master/src/main/python/apache/thermos/runner/thermos_runner.py).

Instead of passing this as a command line arg, we could just:

from twitter.common.log.options import LogOptions

LogOptions.set_simple(True)

(We already do this in our internal fork, no idea why it's done there 
instead of the main project)


- Joshua Cohen


On April 21, 2017, 9:35 a.m., Stephan Erb wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58609/
> ---
> 
> (Updated April 21, 2017, 9:35 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Switch Thermos runner to simple disk log layout.
> 
> This collapses the individual files per loglevel. We will therefore have just 
> one log file for executor and one for the runner.
> 
> -- 
> 
> Not to reviwers: I am unsure why the executor log file defaults to __main__ 
> rather than the name of the binary. I decided not to dig deeper here, as it 
> is not my main problem during operations.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/executor/thermos_task_runner.py 
> 8f88af4c24ddc603fa12587741af56a6c711e420 
> 
> 
> Diff: https://reviews.apache.org/r/58609/diff/1/
> 
> 
> Testing
> ---
> 
> ./pants test.pytest src/test/python::
> 
> 
> File Attachments
> 
> 
> after.png
>   
> https://reviews.apache.org/media/uploaded/files/2017/04/21/8816f094-b473-45bd-8d9a-fd2f3fad7dfe__after.png
> before.png
>   
> https://reviews.apache.org/media/uploaded/files/2017/04/21/f10b3500-a66b-4850-a571-f964b97b7b40__before.png
> 
> 
> Thanks,
> 
> Stephan Erb
> 
>

Re: Review Request 58609: Switch Thermos runner to simple disk log layout

[Aurora ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58609/#review172645
---



Master (b847db8) is red with this patch.
  ./build-support/jenkins/build.sh

  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$8
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$7
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$9
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$4
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$3
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$6
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$5
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$2
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$1
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/LogStorage$Settings
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/LogStorage$ScheduledExecutorSchedulingService
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/LogStorageModule
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$SnapshotField
  Test coverage missing for 
org/apache/aurora/scheduler/storage/backup/TemporaryStorage$TemporaryStorageFactory$1
  Test coverage missing for 
org/apache/aurora/scheduler/storage/backup/BackupModule
  Test coverage missing for 
org/apache/aurora/scheduler/storage/backup/Recovery$RecoveryImpl
  Test coverage missing for 
org/apache/aurora/scheduler/storage/backup/TemporaryStorage$TemporaryStorageFactory
  Test coverage missing for 
org/apache/aurora/scheduler/storage/backup/Recovery$RecoveryImpl$PendingRecovery
  Test coverage missing for org/apache/aurora/scheduler/TaskVars
  Test coverage missing for 
org/apache/aurora/scheduler/SchedulerLifecycle$DefaultDelayedActions
  Test coverage missing for 
org/apache/aurora/scheduler/TierManager$TierManagerImpl$TierConfig
  Test coverage missing for org/apache/aurora/scheduler/TaskVars$Counter
  Test coverage missing for org/apache/aurora/scheduler/TaskVars$1
  Test coverage missing for 
org/apache/aurora/scheduler/SchedulerModule$TaskEventBatchWorker
  Test coverage missing for org/apache/aurora/scheduler/HostOffer$1
  Test coverage missing for org/apache/aurora/scheduler/SchedulerModule
  Test coverage missing for 
org/apache/aurora/scheduler/TaskIdGenerator$TaskIdGeneratorImpl
  Test coverage missing for org/apache/aurora/scheduler/SchedulerModule$1
  Test coverage missing for org/apache/aurora/scheduler/TaskStatusHandlerImpl
  Test coverage missing for org/apache/aurora/scheduler/TaskStatusHandlerImpl$1
  Test coverage missing for org/apache/aurora/scheduler/TierModule
  Test coverage missing for 
org/apache/aurora/scheduler/storage/db/typehandlers/VolumeModeTypeHandler

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
==

BUILD FAILED

Total time: 8 mins 14.05 secs


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On April 21, 2017, 11:35 a.m., Stephan Erb wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58609/
> ---
> 
> (Updated April 21, 2017, 11:35 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Switch Thermos runner to simple disk log layout.
> 
> This collapses the individual files per loglevel. We will therefore have just 
> one log file for executor and one for the runner.
> 
> -- 
> 
> Not to reviwers: I am unsure why the executor log file defaults to __main__ 
> rather than the name of the binary. I decided not to dig deeper here, as it 
> is not my main problem during operations.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/executor/thermos_task_runner.py 
> 8f88af4c24ddc603fa12587741af56a6c711e420 
> 
> 
> Diff: https://reviews.apache.org/r/58609/diff/1/
> 
> 
> Testing
> ---
> 
> ./pants test.pytest src/test/python::
> 
> 
> File Attachments
> 
> 
> after.png
>   
> https://reviews.apache.org/media/uploaded/files/2017/04/21/8816f094-b473-45bd-8d9a-fd2f3fad7dfe__after.png
> before.png
>   
> https://reviews.apache.org/media/uploaded/files/2017/04/21/f10b3500-a66b-4850-a571-f964b97b7b40__before.png
> 
> 
> Thanks,
> 
> Stephan Erb
> 
>