Hi Julien, I double-checked the envs and we don't have such an environment
On Tuesday, June 30, 2020 at 10:04:19 PM UTC+4:30, Julien Pivotto wrote: > > Can you check if you have leftover JAEGER environment variables by any > chance? > > On 30 Jun 09:57, [email protected] wrote: > > Hi, > > We have a Prometheus server which is our primary Prometheus server. we > had > > a Prometheus federation which used to gather all the metrics from the > main > > Prometheus every 90 seconds. we now decided to stop using federation as > the > > backup solution and move the Prometheus configuration to the ansible so > > both servers can have the same configuration at the same time, the main > > Prometheus and the federation's version was 2.15.2. after changing the > > federation server's configuration and putting the same configuration as > the > > primary Prometheus on it, I decided to update the binary file to 2.19.2. > > At first, I got service unavailable alert for about 10 minutes, after an > > hour, we start to get lots of alerts claiming exporter has no data. > > The problem got fixed after 30 minutes but again we start to get lots > of > > alerts about it after few hours. > > when I checked the logs, I saw all jobs have "contecxt deadline > exceeded" > > alert (during the problem I couldn't connect to the web interface as > well) > > whereas I don't get any alert from the primary Prometheus and everything > > works fine there. > > here is my systemd configuration for the secondary Prometheus: > > > > <pre style='color:#000000;background:#ffffff;'><span > style='color:#808030; > > '>[</span><span style='color:#0000e6; '>Unit</span><span > > style='color:#808030; '>]</span> > > <span style='color:#797997; '>Description</span><span > style='color:#808030; > > '>=</span>Prometheus > > <span style='color:#797997; '>After</span><span style='color:#808030; > > '>=</span>network-online<span style='color:#800000; font-weight:bold; > > '>.</span>target > > > > <span style='color:#808030; '>[</span><span style='color:#0000e6; > > '>Service</span><span style='color:#808030; '>]</span> > > <span style='color:#797997; '>Type</span><span style='color:#808030; > > '>=</span>simple > > <span style='color:#797997; '>Environment</span><span > style='color:#808030; > > '>=</span><span style='color:#0000e6; '>"</span><span > style='color:#797997; > > '>GOMAXPROCS</span><span style='color:#808030; '>=</span><span > > style='color:#0000e6; '>8"</span> > > <span style='color:#797997; '>User</span><span style='color:#808030; > > '>=</span>prometheus > > <span style='color:#797997; '>Group</span><span style='color:#808030; > > '>=</span>prometheus > > <span style='color:#797997; '>ExecReload</span><span > style='color:#808030; > > '>=</span><span style='color:#40015a; '>/bin/kill</span> -HUP <span > > style='color:#797997; '>$MAINPID</span> > > <span style='color:#797997; '>ExecStart</span><span > style='color:#808030; > > '>=</span><span style='color:#40015a; > '>/usr/local/sbin/prometheus</span> > > <span style='color:#0f69ff; '>\</span> > > --<span style='color:#797997; '>config.file</span><span > > style='color:#808030; '>=</span><span style='color:#40015a; > > '>/etc/prometheus/prometheus.yml</span> <span style='color:#0f69ff; > > '>\</span> > > --<span style='color:#797997; '>storage.tsdb.path</span><span > > style='color:#808030; '>=</span><span style='color:#40015a; > > '>/var/lib/prometheus</span> <span style='color:#0f69ff; '>\</span> > > --<span style='color:#797997; > '>storage.tsdb.retention.time</span><span > > style='color:#808030; '>=</span>30d <span style='color:#0f69ff; > '>\</span> > > --<span style='color:#797997; > '>storage.tsdb.retention.size</span><span > > style='color:#808030; '>=</span>275GB <span style='color:#0f69ff; > '>\</span> > > --<span style='color:#797997; '>web.console.libraries</span><span > > style='color:#808030; '>=</span><span style='color:#40015a; > > '>/etc/prometheus/console_libraries</span> <span style='color:#0f69ff; > > '>\</span> > > --<span style='color:#797997; '>web.console.templates</span><span > > style='color:#808030; '>=</span><span style='color:#40015a; > > '>/etc/prometheus/consoles</span> <span style='color:#0f69ff; '>\</span> > > --web<span style='color:#800000; font-weight:bold; '>.</span><span > > style='color:#bb7977; font-weight:bold; '>enable</span>-admin-api <span > > style='color:#0f69ff; '>\</span> > > --<span style='color:#797997; '>web.listen-address</span><span > > style='color:#808030; '>=</span><span style='color:#008c00; > '>0</span><span > > style='color:#800000; font-weight:bold; '>.</span><span > > style='color:#008c00; '>0</span><span style='color:#800000; > > font-weight:bold; '>.</span><span style='color:#008c00; '>0</span><span > > style='color:#800000; font-weight:bold; '>.</span><span > > style='color:#008c00; '>0</span><span style='color:#808030; > '>:</span><span > > style='color:#008c00; '>9090</span> > > > > <span style='color:#797997; '>CapabilityBoundingSet</span><span > > style='color:#808030; '>=</span>CAP_SET_UID > > <span style='color:#797997; '>LimitNOFILE</span><span > style='color:#808030; > > '>=</span><span style='color:#008c00; '>65000</span> > > <span style='color:#797997; '>LockPersonality</span><span > > style='color:#808030; '>=</span><span style='color:#44aadd; > '>true</span> > > <span style='color:#797997; '>NoNewPrivileges</span><span > > style='color:#808030; '>=</span><span style='color:#44aadd; > '>true</span> > > <span style='color:#797997; '>MemoryDenyWriteExecute</span><span > > style='color:#808030; '>=</span><span style='color:#44aadd; > '>true</span> > > <span style='color:#797997; '>PrivateDevices</span><span > > style='color:#808030; '>=</span><span style='color:#44aadd; > '>true</span> > > <span style='color:#797997; '>PrivateTmp</span><span > style='color:#808030; > > '>=</span><span style='color:#44aadd; '>true</span> > > <span style='color:#797997; '>ProtectHome</span><span > style='color:#808030; > > '>=</span><span style='color:#44aadd; '>true</span> > > <span style='color:#797997; '>RemoveIPC</span><span > style='color:#808030; > > '>=</span><span style='color:#44aadd; '>true</span> > > <span style='color:#797997; '>RestrictSUIDSGID</span><span > > style='color:#808030; '>=</span><span style='color:#44aadd; > '>true</span> > > <span style='color:#797997; '>CPUAccounting</span><span > > style='color:#808030; '>=</span>yes > > <span style='color:#797997; '>MemoryAccounting</span><span > > style='color:#808030; '>=</span>yes > > <span style='color:#696969; '>#SystemCallFilter=@signal @timer</span> > > > > <span style='color:#797997; '>ReadWritePaths</span><span > > style='color:#808030; '>=</span><span style='color:#40015a; > > '>/var/lib/prometheus</span> > > > > <span style='color:#797997; '>PrivateUsers</span><span > > style='color:#808030; '>=</span><span style='color:#44aadd; > '>true</span> > > <span style='color:#797997; '>ProtectControlGroups</span><span > > style='color:#808030; '>=</span><span style='color:#44aadd; > '>true</span> > > <span style='color:#797997; '>ProtectKernelModules</span><span > > style='color:#808030; '>=</span><span style='color:#44aadd; > '>true</span> > > <span style='color:#797997; '>ProtectKernelTunables</span><span > > style='color:#808030; '>=</span><span style='color:#44aadd; > '>true</span> > > <span style='color:#797997; '>ProtectSystem</span><span > > style='color:#808030; '>=</span>strict > > > > > > <span style='color:#797997; '>SyslogIdentifier</span><span > > style='color:#808030; '>=</span>prometheus > > <span style='color:#797997; '>Restart</span><span style='color:#808030; > > '>=</span>always > > > > <span style='color:#808030; '>[</span><span style='color:#0000e6; > > '>Install</span><span style='color:#808030; '>]</span> > > <span style='color:#797997; '>WantedBy</span><span style='color:#808030; > > '>=</span>multi-user<span style='color:#800000; font-weight:bold; > > '>.</span>target > > </pre> > > <!--Created using ToHtml.com on 2020-06-30 16:52:20 UTC --> > > > > -- > > You received this message because you are subscribed to the Google > Groups "Prometheus Users" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected] <javascript:>. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/5362956f-268d-477e-9d9b-0146e73f2fden%40googlegroups.com. > > > > > -- > Julien Pivotto > @roidelapluie > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/819eead2-969a-4552-83b2-ba454b6d9cd2o%40googlegroups.com.
