Looks like a memory issue. How much RAM you have on the server? https://github.com/prometheus/prometheus/issues/4945
On Wed, Apr 15, 2020, 11:09 AM Ben Kochie <[email protected]> wrote: > That looks like an incomplete stack trace. Can you get the full stack > trace from your syslog? > > There's not enough information to go on here. Please mention what version > of Prometheus this is, and include your prometheus.yml. > > On Wed, Apr 15, 2020 at 5:49 PM Pooja Chauhan <[email protected]> > wrote: > >> Hi , >> For the past 2 days after restart of prometheus ,it stops in few hours. >> >> output of journalctl command is : >> >> -- Logs begin at Wed 2020-03-11 05:15:30 UTC, end at Wed 2020-04-15 >> 15:40:24 UTC. -- >> Apr 15 14:23:38 ip-172-28-48-25 systemd[1]: prometheus.service: Failed >> with result 'exit-code'. >> Apr 15 14:23:38 ip-172-28-48-25 systemd[1]: prometheus.service: Main >> process exited, code=exited, status=2/INVALIDARGUMENT >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by >> github.com/prometheus/prometheus/scrape.(*scrapePool).sync >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> /app/scrape/scrape.go:854 +0x822 >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85e80, >> 0x14f46b0400, 0x6fc23ac00, 0x0) >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: goroutine 573 [select]: >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> /app/scrape/scrape.go:336 +0x45d >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by >> github.com/prometheus/prometheus/scrape.(*scrapePool).sync >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> /app/scrape/scrape.go:774 +0x113 >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85d80, >> 0x14f46b0400, 0x6fc23ac00, 0x0) >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: goroutine 572 [select, >> 1 minutes]: >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> /app/scrape/scrape.go:336 +0x45d >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by >> github.com/prometheus/prometheus/scrape.(*scrapePool).sync >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> /app/scrape/scrape.go:774 +0x113 >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85c80, >> 0x14f46b0400, 0x6fc23ac00, 0x0) >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: goroutine 571 [select, >> 1 minutes]: >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> /app/scrape/scrape.go:336 +0x45d >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by >> github.com/prometheus/prometheus/scrape.(*scrapePool).sync >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> /app/scrape/scrape.go:813 +0x487 >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85b80, >> 0x14f46b0400, 0x6fc23ac00, 0x0) >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> /app/scrape/scrape.go:471 +0x111 >> >> >> and ubuntu@ip-172-28-48-25:/etc/systemd/system$ sudo service prometheus >> status >> ● prometheus.service - Prometheus Server >> Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; >> vendor preset: enabled) >> Active: failed (Result: exit-code) since Wed 2020-04-15 14:23:38 UTC; >> 1h 25min ago >> Docs: https://prometheus.io/docs/introduction/overview/ >> Process: 6138 ExecStart=/etc/prometheus/prometheus >> --config.file=/etc/prometheus/prometheus.yml >> --storage.tsdb.path=/var/lib/prometheus --storage.tsdb.retention=7d (c >> Main PID: 6138 (code=exited, status=2) >> >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85d80, >> 0x14f46b0400, 0x6fc23ac00, 0x0) >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> /app/scrape/scrape.go:774 +0x113 >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by >> github.com/prometheus/prometheus/scrape.(*scrapePool).sync >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> /app/scrape/scrape.go:336 +0x45d >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: goroutine 573 [select]: >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85e80, >> 0x14f46b0400, 0x6fc23ac00, 0x0) >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: >> /app/scrape/scrape.go:854 +0x822 >> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by >> github.com/prometheus/prometheus/scrape.(*scrapePool).sync >> Apr 15 14:23:38 ip-172-28-48-25 systemd[1]: prometheus.service: Main >> process exited, code=exited, status=2/INVALIDA >> >> >> >> >> Please suggest how to resolve this >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/d6d0c130-0bf6-4fde-b4a2-117fce042f6e%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-users/d6d0c130-0bf6-4fde-b4a2-117fce042f6e%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/CABbyFmo2vxZpVy2xzgkHpgCU151Q4Jjd4Z_fh17fTE8PrwEwHw%40mail.gmail.com > <https://groups.google.com/d/msgid/prometheus-users/CABbyFmo2vxZpVy2xzgkHpgCU151Q4Jjd4Z_fh17fTE8PrwEwHw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAKimyZtHwsQq1ZbFJWBPoWe28bC8pAMrir%2BjXaykXkaxzEXbcg%40mail.gmail.com.
