Harpreet created MESOS-5884:
-------------------------------
Summary: Internal virtual files path map does not update to
reflect GLOG rotation
Key: MESOS-5884
URL: https://issues.apache.org/jira/browse/MESOS-5884
Project: Mesos
Issue Type: Bug
Affects Versions: 0.28.1
Environment: CoreOS 766.5.0, Mesos 0.28.1 (DC/OS 1.7.2)
Reporter: Harpreet
Priority: Minor
I was trying to access Mesos Master logs on a log running cluster from the
Mesos UI and was greeted with {{(FAILED TO INITIALIZE...RETRYING)}} in the log
pailer.
To debug this I hit {{/file/read?path=/master/log}} endpoint and was greeted
with {{Failed to open file at 'Failed to open file at
'/var/log/mesos/mesos-master.<hostname>.invalid-user.log.INFO.20160623-082236.807':
No such file or directory.}}
I ssh-ed into the node and checked, sure enough there was no file named
{{/var/log/mesos/mesos-master.<hostname>.invalid-user.log.INFO.20160623-082236.807}},
but there was
{{/var/log/mesos/mesos-master.<hostname>.invalid-user.log.INFO.20160719-060435.807}}
where the newest log lines were going. This made me suspect log rotation.
I confirmed using {{/files/debug}} to make sure that the virtual path mapping
for {{/master/log}} was stale.
This was a production cluster so I did not mess with it anymore since I could
access the logs by ssh-ing into the box anyways. I created another cluster with
same version of DC/OS to experiment with. This is what I did to confirm my
suspicion :
# Spun up a DC/OS 1.7.2 cluster
# Stopped `dcos-logrotate.timer` on the leading master to eliminate this as a
factor/variable
# Confirmed everything worked as expected by :
## accessing master logs from Mesos UI to verify logs streaming in
## hitting {{/files/read?path=/master/log}}
## matching {{/files/debug}} output with actual file name for logs on the box
# Flooded the master logs (using curl requests) to initiate GLOG rotation
# After GLOG rotation verified that file path mapping was not updated by
## hitting {{/files/debug}} to confirm mapping was stale
## hitting {{/files/read?path=/master/log}} lead to {{No such file or
directory. error}}
## trying to access master logs from Mesos UI showed last lines from older log
file rather than streaming logs. This file would have eventually gotten deleted
by {{dcos-logrotate}} and lead to {{(FAILED TO INITIALIZE...RETRYING)}}
# Restarted mesos master process to notice that the mapping is updated and the
logs are accessible again from the Mesos UI
So the impact of internal virtual files path map not updating is that you would
either see stale master logs or no master logs from the Mesos UI. Master
process restart fixes the mapping until its broken again by GLOG rotation
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
