Can you check if you have leftover JAEGER environment variables by any chance?
On 30 Jun 09:57, [email protected] wrote: > Hi, > We have a Prometheus server which is our primary Prometheus server. we had > a Prometheus federation which used to gather all the metrics from the main > Prometheus every 90 seconds. we now decided to stop using federation as the > backup solution and move the Prometheus configuration to the ansible so > both servers can have the same configuration at the same time, the main > Prometheus and the federation's version was 2.15.2. after changing the > federation server's configuration and putting the same configuration as the > primary Prometheus on it, I decided to update the binary file to 2.19.2. > At first, I got service unavailable alert for about 10 minutes, after an > hour, we start to get lots of alerts claiming exporter has no data. > The problem got fixed after 30 minutes but again we start to get lots of > alerts about it after few hours. > when I checked the logs, I saw all jobs have "contecxt deadline exceeded" > alert (during the problem I couldn't connect to the web interface as well) > whereas I don't get any alert from the primary Prometheus and everything > works fine there. > here is my systemd configuration for the secondary Prometheus: > > <pre style='color:#000000;background:#ffffff;'><span style='color:#808030; > '>[</span><span style='color:#0000e6; '>Unit</span><span > style='color:#808030; '>]</span> > <span style='color:#797997; '>Description</span><span style='color:#808030; > '>=</span>Prometheus > <span style='color:#797997; '>After</span><span style='color:#808030; > '>=</span>network-online<span style='color:#800000; font-weight:bold; > '>.</span>target > > <span style='color:#808030; '>[</span><span style='color:#0000e6; > '>Service</span><span style='color:#808030; '>]</span> > <span style='color:#797997; '>Type</span><span style='color:#808030; > '>=</span>simple > <span style='color:#797997; '>Environment</span><span style='color:#808030; > '>=</span><span style='color:#0000e6; '>"</span><span style='color:#797997; > '>GOMAXPROCS</span><span style='color:#808030; '>=</span><span > style='color:#0000e6; '>8"</span> > <span style='color:#797997; '>User</span><span style='color:#808030; > '>=</span>prometheus > <span style='color:#797997; '>Group</span><span style='color:#808030; > '>=</span>prometheus > <span style='color:#797997; '>ExecReload</span><span style='color:#808030; > '>=</span><span style='color:#40015a; '>/bin/kill</span> -HUP <span > style='color:#797997; '>$MAINPID</span> > <span style='color:#797997; '>ExecStart</span><span style='color:#808030; > '>=</span><span style='color:#40015a; '>/usr/local/sbin/prometheus</span> > <span style='color:#0f69ff; '>\</span> > --<span style='color:#797997; '>config.file</span><span > style='color:#808030; '>=</span><span style='color:#40015a; > '>/etc/prometheus/prometheus.yml</span> <span style='color:#0f69ff; > '>\</span> > --<span style='color:#797997; '>storage.tsdb.path</span><span > style='color:#808030; '>=</span><span style='color:#40015a; > '>/var/lib/prometheus</span> <span style='color:#0f69ff; '>\</span> > --<span style='color:#797997; '>storage.tsdb.retention.time</span><span > style='color:#808030; '>=</span>30d <span style='color:#0f69ff; '>\</span> > --<span style='color:#797997; '>storage.tsdb.retention.size</span><span > style='color:#808030; '>=</span>275GB <span style='color:#0f69ff; '>\</span> > --<span style='color:#797997; '>web.console.libraries</span><span > style='color:#808030; '>=</span><span style='color:#40015a; > '>/etc/prometheus/console_libraries</span> <span style='color:#0f69ff; > '>\</span> > --<span style='color:#797997; '>web.console.templates</span><span > style='color:#808030; '>=</span><span style='color:#40015a; > '>/etc/prometheus/consoles</span> <span style='color:#0f69ff; '>\</span> > --web<span style='color:#800000; font-weight:bold; '>.</span><span > style='color:#bb7977; font-weight:bold; '>enable</span>-admin-api <span > style='color:#0f69ff; '>\</span> > --<span style='color:#797997; '>web.listen-address</span><span > style='color:#808030; '>=</span><span style='color:#008c00; '>0</span><span > style='color:#800000; font-weight:bold; '>.</span><span > style='color:#008c00; '>0</span><span style='color:#800000; > font-weight:bold; '>.</span><span style='color:#008c00; '>0</span><span > style='color:#800000; font-weight:bold; '>.</span><span > style='color:#008c00; '>0</span><span style='color:#808030; '>:</span><span > style='color:#008c00; '>9090</span> > > <span style='color:#797997; '>CapabilityBoundingSet</span><span > style='color:#808030; '>=</span>CAP_SET_UID > <span style='color:#797997; '>LimitNOFILE</span><span style='color:#808030; > '>=</span><span style='color:#008c00; '>65000</span> > <span style='color:#797997; '>LockPersonality</span><span > style='color:#808030; '>=</span><span style='color:#44aadd; '>true</span> > <span style='color:#797997; '>NoNewPrivileges</span><span > style='color:#808030; '>=</span><span style='color:#44aadd; '>true</span> > <span style='color:#797997; '>MemoryDenyWriteExecute</span><span > style='color:#808030; '>=</span><span style='color:#44aadd; '>true</span> > <span style='color:#797997; '>PrivateDevices</span><span > style='color:#808030; '>=</span><span style='color:#44aadd; '>true</span> > <span style='color:#797997; '>PrivateTmp</span><span style='color:#808030; > '>=</span><span style='color:#44aadd; '>true</span> > <span style='color:#797997; '>ProtectHome</span><span style='color:#808030; > '>=</span><span style='color:#44aadd; '>true</span> > <span style='color:#797997; '>RemoveIPC</span><span style='color:#808030; > '>=</span><span style='color:#44aadd; '>true</span> > <span style='color:#797997; '>RestrictSUIDSGID</span><span > style='color:#808030; '>=</span><span style='color:#44aadd; '>true</span> > <span style='color:#797997; '>CPUAccounting</span><span > style='color:#808030; '>=</span>yes > <span style='color:#797997; '>MemoryAccounting</span><span > style='color:#808030; '>=</span>yes > <span style='color:#696969; '>#SystemCallFilter=@signal @timer</span> > > <span style='color:#797997; '>ReadWritePaths</span><span > style='color:#808030; '>=</span><span style='color:#40015a; > '>/var/lib/prometheus</span> > > <span style='color:#797997; '>PrivateUsers</span><span > style='color:#808030; '>=</span><span style='color:#44aadd; '>true</span> > <span style='color:#797997; '>ProtectControlGroups</span><span > style='color:#808030; '>=</span><span style='color:#44aadd; '>true</span> > <span style='color:#797997; '>ProtectKernelModules</span><span > style='color:#808030; '>=</span><span style='color:#44aadd; '>true</span> > <span style='color:#797997; '>ProtectKernelTunables</span><span > style='color:#808030; '>=</span><span style='color:#44aadd; '>true</span> > <span style='color:#797997; '>ProtectSystem</span><span > style='color:#808030; '>=</span>strict > > > <span style='color:#797997; '>SyslogIdentifier</span><span > style='color:#808030; '>=</span>prometheus > <span style='color:#797997; '>Restart</span><span style='color:#808030; > '>=</span>always > > <span style='color:#808030; '>[</span><span style='color:#0000e6; > '>Install</span><span style='color:#808030; '>]</span> > <span style='color:#797997; '>WantedBy</span><span style='color:#808030; > '>=</span>multi-user<span style='color:#800000; font-weight:bold; > '>.</span>target > </pre> > <!--Created using ToHtml.com on 2020-06-30 16:52:20 UTC --> > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/5362956f-268d-477e-9d9b-0146e73f2fden%40googlegroups.com. -- Julien Pivotto @roidelapluie -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20200630173407.GA844212%40oxygen.
