Hi, I have attached the data in *prometheus_rate_1h.txt.*
*Also, see the graph.*
[image: promethesu-rate-1hr.png]
On Tuesday, April 14, 2020 at 12:30:58 AM UTC+5:30, Julien Pivotto wrote:
>
> On 13 Apr 11:56, Shubham Choudhary wrote:
> > I am using the Prometheus recording rule to capture the CPU usage over 5
> > mins and later want to use it to get maximum CPU used for 5 mins in the
> > last 30 days.
> >
> > Recording Rule
> >
> > - expr: 1 - avg by (instance)
> > > (rate(node_cpu_seconds_total{mode="idle"}[5m]))
> > > record: instance:node_cpu_usage:rate5m
> >
> >
> > Max. CPU used in last 30 days over 5 mins
> >
> > max_over_time(instance:node_cpu_usage:rate5m[30d])
> >
> >
> > Now, let's imagine a situation where Prometheus starts to pull data with
> an
> > interval of 15sec which started at 00:00:00 so
> >
> > 1. 1st pull was at 00:00:00. Here, the instance:node_cpu_usage:rate5m
> is
> > not calculated since the rate function needs a minimum of 2 data
> points.
> > 2. 2nd pull was at 00:00:15. *Here, the
> instance:node_cpu_usage:rate5m
> > is calculated as the difference of 2nd pull - 1st pull data divided
> by 300
> > seconds. Now, this is the issue of why rate function dividing by 300
> > instead of 15.*
> > 3. 3rd pull was at 00:00:45. Again, the same scenario occurs.
> > 4. The wrong data is being saved in instance:node_cpu_usage:rate5m
> till
> > 5th minute.
>
>
> Please provide the raw data:
>
> node_cpu_seconds_total{mode="idle"}[1h] in the "table" view.
>
> Thanks.
>
> >
> > [image: prometheus_rate_less_data.png]
> >
> >
> > *So, doing a max_over_time(instance:node_cpu_usage:rate5m[30d]) will for
> > sure give me the first value which is wrong.*
> > Also if there are server crashes or network issue then the rate gives
> wrong
> > data.
> >
> > How can I overcome this outlier?
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to [email protected] <javascript:>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/995498ad-5ac2-4b97-a53f-0ae3f1e97c18%40googlegroups.com.
>
>
>
>
>
> --
> (o- Julien Pivotto
> //\ Open-Source Consultant
> V_/_ Inuits - https://www.inuits.eu
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/139b857d-1a3d-47f8-8b6a-5591a4593672%40googlegroups.com.
Element Value
node_cpu_seconds_total{cpu="0",instance="XXXX:9100",mode="idle"}
20653667.85 @1586801414.34
20653682.55 @1586801429.34
20653697.28 @1586801444.34
20653712.01 @1586801459.34
20653726.75 @1586801474.34
20653741.48 @1586801489.34
20653756.18 @1586801504.34
20653770.92 @1586801519.34
20653785.65 @1586801534.34
20653800.39 @1586801549.34
20653815.12 @1586801564.34
20654287.02 @1586802044.34
20654301.7 @1586802059.34
20654316.4 @1586802074.34
20654331.11 @1586802089.34
20654341.75 @1586802104.34
20654356.49 @1586802119.34
20654371.17 @1586802134.34
20654385.89 @1586802149.34
20654400.62 @1586802164.34
20654415.37 @1586802179.34
20654430.14 @1586802194.34
20654444.87 @1586802209.34
20654459.63 @1586802224.341
20654474.38 @1586802239.34
20654489.14 @1586802254.34
20654503.88 @1586802269.34
20654518.64 @1586802284.34
20654533.39 @1586802299.34
20654548.13 @1586802314.34
20654562.87 @1586802329.34
20654577.63 @1586802344.34
20654592.38 @1586802359.34
20654607.12 @1586802374.34
20654621.87 @1586802389.34
20654636.64 @1586802404.34
20654651.39 @1586802419.34
20654666.16 @1586802434.34
20654680.93 @1586802449.34
20654695.68 @1586802464.34
20654710.43 @1586802479.34
20654725.18 @1586802494.34
20654739.93 @1586802509.34
20654754.68 @1586802524.34
20654769.44 @1586802539.34
20654784.19 @1586802554.34
20654798.95 @1586802569.34
20654813.69 @1586802584.34
20654828.43 @1586802599.34
20654842 @1586802614.34
20654856.64 @1586802629.34
20654871.33 @1586802644.34
20654886.02 @1586802659.34
20654900.75 @1586802674.34
20654915.46 @1586802689.34
20654930.18 @1586802704.34
20654944.91 @1586802719.34
20654959.62 @1586802734.34
20654974.35 @1586802749.34
20654989.06 @1586802764.34
20655003.79 @1586802779.34
20655018.52 @1586802794.34
20655033.27 @1586802809.34
20655048.01 @1586802824.34
20655062.74 @1586802839.34
20655077.5 @1586802854.34
20655092.27 @1586802869.34
20655107.03 @1586802884.34
20655121.78 @1586802899.34
20655136.52 @1586802914.34
20655151.24 @1586802929.34
20655165.99 @1586802944.34
20655180.72 @1586802959.34
20655195.49 @1586802974.34
20655210.23 @1586802989.34
20655224.95 @1586803004.34
20655239.7 @1586803019.34
20655254.43 @1586803034.34
20655269.17 @1586803049.34
20655283.92 @1586803064.34
20655298.69 @1586803079.34
20655313.43 @1586803094.34
20655328.18 @1586803109.34
20655342.96 @1586803124.34
20655357.72 @1586803139.34
20655372.44 @1586803154.34
20655386.18 @1586803169.34
20655397.94 @1586803184.34
20655412.67 @1586803199.34
20655427.37 @1586803214.34
20655442.05 @1586803229.34
20655456.74 @1586803244.34
20655471.46 @1586803259.34
20655486.18 @1586803274.34
20655500.89 @1586803289.34
20655515.63 @1586803304.34
20655530.36 @1586803319.34
20655545.07 @1586803334.34
20655559.78 @1586803349.34
20655574.51 @1586803364.34
20655589.27 @1586803379.34
20655604 @1586803394.34
20655618.74 @1586803409.34
20655633.47 @1586803424.34
20655648.22 @1586803439.34
20655662.99 @1586803454.34
20655677.74 @1586803469.34
20655692.5 @1586803484.34
20655707.25 @1586803499.34
20655721.97 @1586803514.34
20655736.68 @1586803529.34
20655751.4 @1586803544.34
20655766.14 @1586803559.34
20655780.89 @1586803574.34
20655795.65 @1586803589.34
20655810.41 @1586803604.34
20655825.17 @1586803619.34
20655839.9 @1586803634.34
20655854.64 @1586803649.34
20655869.4 @1586803664.34
20655884.15 @1586803679.34
20655898.91 @1586803694.34
20655913.67 @1586803709.34
20655928.44 @1586803724.34
20655943.19 @1586803739.34
20655957.92 @1586803754.34
20655972.67 @1586803769.34
20655987.43 @1586803784.34
20656002.17 @1586803799.34
20656016.87 @1586803814.34
20656031.57 @1586803829.34
20656046.3 @1586803844.34
20656061.02 @1586803859.34
20656075.74 @1586803874.34
20656090.46 @1586803889.34
20656105.16 @1586803904.34
20656119.88 @1586803919.34
20656134.59 @1586803934.34
20656149.31 @1586803949.34
20656164.05 @1586803964.34
20656178.77 @1586803979.34
20656193.54 @1586803994.34
20656208.3 @1586804009.34
20656223.06 @1586804024.34
20656237.81 @1586804039.34
20656252.56 @1586804054.34
20656267.3 @1586804069.34
20656282.05 @1586804084.34
20656296.79 @1586804099.34
20656311.51 @1586804114.34
20656326.25 @1586804129.34
20656338.9 @1586804144.34
20656351.06 @1586804159.34
20656365.8 @1586804174.34
20656380.52 @1586804189.34
20656395.29 @1586804204.34
20656410.06 @1586804219.34
20656424.82 @1586804234.34
20656439.59 @1586804249.34
20656454.35 @1586804264.34
20656469.11 @1586804279.34
20656483.85 @1586804294.34
20656498.61 @1586804309.34
20656513.33 @1586804324.34
20656528.08 @1586804339.34
20656542.81 @1586804354.34
20656557.53 @1586804369.34
20656572.29 @1586804384.34
20656587.04 @1586804399.34
20656601.74 @1586804414.34
20656616.42 @1586804429.34
20656631.06 @1586804444.34
20656645.77 @1586804459.34
20656660.47 @1586804474.34
20656675.17 @1586804489.34
20656689.9 @1586804504.34
20656704.63 @1586804519.34
20656719.34 @1586804534.34
20656734.1 @1586804549.34
20656748.84 @1586804564.34
20656763.59 @1586804579.34
20656778.35 @1586804594.34
20656793.1 @1586804609.34
20656807.83 @1586804624.34
20656822.59 @1586804639.34
20656837.33 @1586804654.34
20656852.08 @1586804669.34
20656866.81 @1586804684.34
20656881.55 @1586804699.34
20656896.29 @1586804714.34
20656911.02 @1586804729.34
20656925.77 @1586804744.34
20656940.51 @1586804759.34
20656955.24 @1586804774.34
20656969.99 @1586804789.34
20656984.76 @1586804804.34
20656999.52 @1586804819.34
20657014.26 @1586804834.34
20657029.02 @1586804849.34
20657043.77 @1586804864.34
20657058.52 @1586804879.34
20657073.27 @1586804894.34
20657088.02 @1586804909.34
20657102.76 @1586804924.34
20657117.51 @1586804939.34
20657132.22 @1586804954.34
20657146.98 @1586804969.34
20657161.74 @1586804984.34
20657176.51 @1586804999.34
node_cpu_seconds_total{cpu="1",instance="XXX:9100",mode="idle"}
20642910.67 @1586801414.34
20642925.36 @1586801429.34
20642940.06 @1586801444.34
20642954.77 @1586801459.34
20642969.5 @1586801474.34
20642984.2 @1586801489.34
20642998.89 @1586801504.34
20643013.6 @1586801519.34
20643028.35 @1586801534.34
20643043.08 @1586801549.34
20643057.82 @1586801564.34
20643529.56 @1586802044.34
20643544.28 @1586802059.34
20643559 @1586802074.34
20643573.71 @1586802089.34
20643583.94 @1586802104.34
20643598.61 @1586802119.34
20643613.29 @1586802134.34
20643627.99 @1586802149.34
20643642.73 @1586802164.34
20643657.46 @1586802179.34
20643672.19 @1586802194.34
20643686.91 @1586802209.34
20643701.65 @1586802224.341
20643716.39 @1586802239.34
20643731.13 @1586802254.34
20643745.87 @1586802269.34
20643760.6 @1586802284.34
20643775.31 @1586802299.34
20643790.05 @1586802314.34
20643804.79 @1586802329.34
20643819.53 @1586802344.34
20643834.26 @1586802359.34
20643849 @1586802374.34
20643863.74 @1586802389.34
20643878.49 @1586802404.34
20643893.2 @1586802419.34
20643907.93 @1586802434.34
20643922.67 @1586802449.34
20643937.4 @1586802464.34
20643952.14 @1586802479.34
20643966.9 @1586802494.34
20643981.66 @1586802509.34
20643996.42 @1586802524.34
20644011.14 @1586802539.34
20644025.87 @1586802554.34
20644040.61 @1586802569.34
20644055.38 @1586802584.34
20644070.12 @1586802599.34
20644083.37 @1586802614.34
20644097.95 @1586802629.34
20644112.57 @1586802644.34
20644127.26 @1586802659.34
20644141.97 @1586802674.34
20644156.67 @1586802689.34
20644171.36 @1586802704.34
20644186.07 @1586802719.34
20644200.76 @1586802734.34
20644215.46 @1586802749.34
20644230.14 @1586802764.34
20644244.87 @1586802779.34
20644259.62 @1586802794.34
20644274.39 @1586802809.34
20644289.16 @1586802824.34
20644303.92 @1586802839.34
20644318.62 @1586802854.34
20644333.37 @1586802869.34
20644348.12 @1586802884.34
20644362.87 @1586802899.34
20644377.59 @1586802914.34
20644392.3 @1586802929.34
20644407.04 @1586802944.34
20644421.8 @1586802959.34
20644436.56 @1586802974.34
20644451.31 @1586802989.34
20644466.05 @1586803004.34
20644480.79 @1586803019.34
20644495.56 @1586803034.34
20644510.31 @1586803049.34
20644525.06 @1586803064.34
20644539.8 @1586803079.34
20644554.53 @1586803094.34
20644569.26 @1586803109.34
20644584 @1586803124.34
20644598.73 @1586803139.34
20644613.46 @1586803154.34
20644627.02 @1586803169.34
20644638.4 @1586803184.34
20644653.13 @1586803199.34
20644667.79 @1586803214.34
20644682.44 @1586803229.34
20644697.14 @1586803244.34
20644711.81 @1586803259.34
20644726.52 @1586803274.34
20644741.17 @1586803289.34
20644755.86 @1586803304.34
20644770.58 @1586803319.34
20644785.29 @1586803334.34
20644799.99 @1586803349.34
20644814.71 @1586803364.34
20644829.45 @1586803379.34
20644844.19 @1586803394.34
20644858.95 @1586803409.34
20644873.69 @1586803424.34
20644888.44 @1586803439.34
20644903.18 @1586803454.34
20644917.94 @1586803469.34
20644932.7 @1586803484.34
20644947.47 @1586803499.34
20644962.19 @1586803514.34
20644976.85 @1586803529.34
20644991.57 @1586803544.34
20645006.29 @1586803559.34
20645021.03 @1586803574.34
20645035.76 @1586803589.34
20645050.53 @1586803604.34
20645065.29 @1586803619.34
20645080.07 @1586803634.34
20645094.8 @1586803649.34
20645109.56 @1586803664.34
20645124.34 @1586803679.34
20645139.08 @1586803694.34
20645153.82 @1586803709.34
20645168.57 @1586803724.34
20645183.32 @1586803739.34
20645198.03 @1586803754.34
20645212.76 @1586803769.34
20645227.51 @1586803784.34
20645242.26 @1586803799.34
20645256.9 @1586803814.34
20645271.59 @1586803829.34
20645286.29 @1586803844.34
20645301 @1586803859.34
20645315.73 @1586803874.34
20645330.47 @1586803889.34
20645345.18 @1586803904.34
20645359.85 @1586803919.34
20645374.56 @1586803934.34
20645389.25 @1586803949.34
20645403.96 @1586803964.34
20645418.65 @1586803979.34
20645433.39 @1586803994.34
20645448.15 @1586804009.34
20645462.89 @1586804024.34
20645477.64 @1586804039.34
20645492.36 @1586804054.34
20645507.09 @1586804069.34
20645521.82 @1586804084.34
20645536.56 @1586804099.34
20645551.29 @1586804114.34
20645566.02 @1586804129.34
20645579.16 @1586804144.34
20645591.66 @1586804159.34
20645606.36 @1586804174.34
20645621.1 @1586804189.34
20645635.87 @1586804204.34
20645650.61 @1586804219.34
20645665.35 @1586804234.34
20645680.1 @1586804249.34
20645694.84 @1586804264.34
20645709.57 @1586804279.34
20645724.33 @1586804294.34
20645739.07 @1586804309.34
20645753.81 @1586804324.34
20645768.55 @1586804339.34
20645783.3 @1586804354.34
20645798.03 @1586804369.34
20645812.79 @1586804384.34
20645827.52 @1586804399.34
20645842.24 @1586804414.34
20645856.91 @1586804429.34
20645871.53 @1586804444.34
20645886.24 @1586804459.34
20645900.88 @1586804474.34
20645915.57 @1586804489.34
20645930.28 @1586804504.34
20645944.99 @1586804519.34
20645959.68 @1586804534.34
20645974.39 @1586804549.34
20645989.11 @1586804564.34
20646003.81 @1586804579.34
20646018.55 @1586804594.34
20646033.31 @1586804609.34
20646048.06 @1586804624.34
20646062.79 @1586804639.34
20646077.53 @1586804654.34
20646092.28 @1586804669.34
20646107.03 @1586804684.34
20646121.8 @1586804699.34
20646136.52 @1586804714.34
20646151.21 @1586804729.34
20646165.97 @1586804744.34
20646180.69 @1586804759.34
20646195.42 @1586804774.34
20646210.13 @1586804789.34
20646224.88 @1586804804.34
20646239.66 @1586804819.34
20646254.41 @1586804834.34
20646269.15 @1586804849.34
20646283.9 @1586804864.34
20646298.65 @1586804879.34
20646313.41 @1586804894.34
20646328.16 @1586804909.34
20646342.91 @1586804924.34
20646357.67 @1586804939.34
20646372.39 @1586804954.34
20646387.11 @1586804969.34
20646401.86 @1586804984.34
20646416.61 @1586804999.34
node_cpu_seconds_total{cpu="0",instance="XXX:9100",mode="idle"}
15920925.13 @1586801443.137
15920935.9 @1586801458.137
15920947.71 @1586801473.137
15920955.46 @1586801488.137
15920967.48 @1586801503.137
15920978.74 @1586801518.137
15920990.92 @1586801533.137
15921003.84 @1586801548.137
15921016.22 @1586801563.137
15921402.23 @1586802043.137
15921414.06 @1586802058.137
15921426.36 @1586802073.138
15921439.06 @1586802088.137
15921449.77 @1586802103.137
15921461.78 @1586802118.138
15921473.73 @1586802133.137
15921484.11 @1586802148.137
15921497.04 @1586802163.138
15921509.73 @1586802178.138
15921521.72 @1586802193.138
15921532.95 @1586802208.137
15921545.57 @1586802223.138
15921557.36 @1586802238.138
15921568.67 @1586802253.138
15921581.63 @1586802268.137
15921593.66 @1586802283.138
15921606.18 @1586802298.138
15921617.84 @1586802313.138
15921629.43 @1586802328.138
15921642.44 @1586802343.138
15921654.49 @1586802358.138
15921666.63 @1586802373.138
15921679.1 @1586802388.138
15921692 @1586802403.138
15921699.97 @1586802418.138
15921708.34 @1586802433.138
15921720.54 @1586802448.138
15921731.43 @1586802463.137
15921744.35 @1586802478.137
15921756.56 @1586802493.138
15921768.75 @1586802508.138
15921780.14 @1586802523.138
15921792.84 @1586802538.138
15921804.67 @1586802553.138
15921816.57 @1586802568.137
15921829.14 @1586802583.138
15921841.18 @1586802598.138
15921851.66 @1586802613.138
15921862.67 @1586802628.138
15921874.53 @1586802643.138
15921887.12 @1586802658.138
15921898.13 @1586802673.138
15921910.22 @1586802688.138
15921922.51 @1586802703.138
15921934.03 @1586802718.138
15921944.09 @1586802733.137
15921956.59 @1586802748.138
15921968.67 @1586802763.137
15921980.12 @1586802778.138
15921991.71 @1586802793.138
15922003.85 @1586802808.138
15922016.15 @1586802823.138
15922027.49 @1586802838.138
15922039.48 @1586802853.137
15922051.88 @1586802868.138
15922063.91 @1586802883.138
15922076.07 @1586802898.138
15922088 @1586802913.138
15922100.27 @1586802928.138
15922112.23 @1586802943.138
15922124.04 @1586802958.137
15922136.08 @1586802973.138
15922145.79 @1586802988.137
15922157.9 @1586803003.138
15922170.47 @1586803018.138
15922182.21 @1586803033.137
15922194.15 @1586803048.138
15922207 @1586803063.137
15922219.35 @1586803078.138
15922230.7 @1586803093.138
15922243.42 @1586803108.137
15922255.39 @1586803123.137
15922267.23 @1586803138.138
15922280.03 @1586803153.137
15922292.41 @1586803168.138
15922305.47 @1586803183.137
15922317.34 @1586803198.137
15922328.98 @1586803213.138
15922340.56 @1586803228.138
15922351.9 @1586803243.138
15922363.94 @1586803258.138
15922375.93 @1586803273.138
15922387.13 @1586803288.138
15922398.42 @1586803303.138
15922410.56 @1586803318.137
15922422.16 @1586803333.138
15922432.13 @1586803348.138
15922444.38 @1586803363.137
15922456.89 @1586803378.137
15922467.15 @1586803393.138
15922474.08 @1586803408.137
15922486.51 @1586803423.138
15922499.25 @1586803438.138
15922510.35 @1586803453.138
15922522.57 @1586803468.138
15922535.22 @1586803483.137
15922547.33 @1586803498.138
15922557.52 @1586803513.138
15922570.26 @1586803528.138
15922582.48 @1586803543.137
15922593.86 @1586803558.138
15922606.32 @1586803573.138
15922618.12 @1586803588.137
15922630.46 @1586803603.137
15922641.11 @1586803618.138
15922653.36 @1586803633.138
15922665.53 @1586803648.138
15922678.17 @1586803663.138
15922689.91 @1586803678.138
15922702.06 @1586803693.138
15922714.79 @1586803708.138
15922725.57 @1586803723.138
15922737.71 @1586803738.138
15922749.96 @1586803753.138
15922761.8 @1586803768.138
15922773.03 @1586803783.138
15922785.81 @1586803798.138
15922798 @1586803813.138
15922808.66 @1586803828.138
15922821.23 @1586803843.138
15922833.89 @1586803858.138
15922845.69 @1586803873.137
15922857.22 @1586803888.138
15922869.62 @1586803903.138
15922881.74 @1586803918.138
15922892.91 @1586803933.138
15922905.04 @1586803948.137
15922917.26 @1586803963.137
15922929.69 @1586803978.138
15922941.32 @1586803993.138
15922953.48 @1586804008.138
15922966.29 @1586804023.138
15922978.35 @1586804038.138
15922989.59 @1586804053.137
15923001.97 @1586804068.138
15923014.63 @1586804083.138
15923025.61 @1586804098.138
15923037.71 @1586804113.138
15923050.17 @1586804128.137
15923060.95 @1586804143.138
15923073.63 @1586804158.138
15923085.8 @1586804173.138
15923097.5 @1586804188.138
15923109.79 @1586804203.138
15923121.96 @1586804218.137
15923134.19 @1586804233.137
15923145.8 @1586804248.138
15923158.63 @1586804263.138
15923171.15 @1586804278.138
15923183.14 @1586804293.138
15923193.79 @1586804308.138
15923206.54 @1586804323.138
15923218.5 @1586804338.138
15923229.97 @1586804353.138
15923236.08 @1586804368.138
15923247.82 @1586804383.137
15923260.59 @1586804398.138
15923271.59 @1586804413.138
15923283.96 @1586804428.138
15923296.26 @1586804443.138
15923308.44 @1586804458.138
15923319.62 @1586804473.137
15923331.03 @1586804488.138
15923343.19 @1586804503.138
15923354.03 @1586804518.137
15923366.45 @1586804533.137
15923378.81 @1586804548.138
15923390.03 @1586804563.138
15923401.2 @1586804578.138
15923413.51 @1586804593.138
15923425.89 @1586804608.138
15923437.46 @1586804623.138
15923449.75 @1586804638.137
15923461.87 @1586804653.138
15923474.14 @1586804668.138
15923486.36 @1586804683.137
15923498.56 @1586804698.138
15923509.94 @1586804713.138
15923520.82 @1586804728.137
15923533.49 @1586804743.138
15923545.7 @1586804758.138
15923557.87 @1586804773.138
15923569.31 @1586804788.138
15923581.04 @1586804803.137
15923594.01 @1586804818.138
15923605.73 @1586804833.138
15923617.79 @1586804848.138
15923630.07 @1586804863.138
15923642.52 @1586804878.138
15923653.53 @1586804893.138
15923665.91 @1586804908.138
15923678.47 @1586804923.137
15923689.84 @1586804938.137
15923702.43 @1586804953.138
15923715.12 @1586804968.138
15923727.87 @1586804983.138
15923739.47 @1586804998.138
15923751.3 @1586805013.137
node_cpu_seconds_total{cpu="1",instance="XXX:9100",mode="idle"}
16067225.91 @1586801428.137
16067238.14 @1586801443.137
16067250.6 @1586801458.137
16067261.95 @1586801473.137
16067268.43 @1586801488.137
16067280.32 @1586801503.137
16067291.32 @1586801518.137
16067303.54 @1586801533.137
16067316.01 @1586801548.137
16067327.36 @1586801563.137
16067709.59 @1586802043.137
16067721.01 @1586802058.137
16067733.1 @1586802073.138
16067745.59 @1586802088.137
16067755.63 @1586802103.137
16067768.43 @1586802118.138
16067780.61 @1586802133.137
16067791.73 @1586802148.137
16067804.3 @1586802163.138
16067816.9 @1586802178.138
16067829.13 @1586802193.138
16067840.51 @1586802208.137
16067853.16 @1586802223.138
16067865.24 @1586802238.138
16067877.47 @1586802253.138
16067889.78 @1586802268.137
16067902.09 @1586802283.138
16067914.83 @1586802298.138
16067926.04 @1586802313.138
16067938.11 @1586802328.138
16067950.72 @1586802343.138
16067962.41 @1586802358.138
16067974.2 @1586802373.138
16067986.71 @1586802388.138
16067999.36 @1586802403.138
16068008.32 @1586802418.138
16068016.96 @1586802433.138
16068029.74 @1586802448.138
16068041.3 @1586802463.137
16068054.04 @1586802478.137
16068066.6 @1586802493.138
16068078.81 @1586802508.138
16068090.6 @1586802523.138
16068103.02 @1586802538.138
16068114.45 @1586802553.138
16068125.94 @1586802568.137
16068138.29 @1586802583.138
16068149.8 @1586802598.138
16068160.7 @1586802613.138
16068171.85 @1586802628.138
16068183.64 @1586802643.138
16068195.99 @1586802658.138
16068207.18 @1586802673.138
16068219.27 @1586802688.138
16068231.45 @1586802703.138
16068243.44 @1586802718.138
16068254.7 @1586802733.137
16068267.32 @1586802748.138
16068279.21 @1586802763.137
16068290.36 @1586802778.138
16068302.17 @1586802793.138
16068313.93 @1586802808.138
16068326.57 @1586802823.138
16068338.2 @1586802838.138
16068350.17 @1586802853.137
16068362.96 @1586802868.138
16068374.87 @1586802883.138
16068387.35 @1586802898.138
16068399.71 @1586802913.138
16068412.02 @1586802928.138
16068422.99 @1586802943.138
16068434.84 @1586802958.137
16068447.07 @1586802973.138
16068457.71 @1586802988.137
16068469.8 @1586803003.138
16068482.27 @1586803018.138
16068493.95 @1586803033.137
16068506.06 @1586803048.138
16068518.36 @1586803063.137
16068530.19 @1586803078.138
16068541.48 @1586803093.138
16068553.95 @1586803108.137
16068566.06 @1586803123.137
16068577.33 @1586803138.138
16068589.69 @1586803153.137
16068601.85 @1586803168.138
16068614.23 @1586803183.137
16068625.15 @1586803198.137
16068636.49 @1586803213.138
16068647.91 @1586803228.138
16068659.19 @1586803243.138
16068671.13 @1586803258.138
16068682.87 @1586803273.138
16068694.63 @1586803288.138
16068706.6 @1586803303.138
16068719.01 @1586803318.137
16068730.93 @1586803333.138
16068741.45 @1586803348.138
16068754.16 @1586803363.137
16068766.73 @1586803378.137
16068776.06 @1586803393.138
16068784.06 @1586803408.137
16068796.27 @1586803423.138
16068808.81 @1586803438.138
16068820.4 @1586803453.138
16068832.45 @1586803468.138
16068844.67 @1586803483.137
16068856.52 @1586803498.138
16068866.94 @1586803513.138
16068879.25 @1586803528.138
16068891.68 @1586803543.137
16068902.7 @1586803558.138
16068914.86 @1586803573.138
16068927.16 @1586803588.137
16068939.84 @1586803603.137
16068952.46 @1586803618.138
16068964.65 @1586803633.138
16068976.79 @1586803648.138
16068989.08 @1586803663.138
16069000.26 @1586803678.138
16069012.61 @1586803693.138
16069024.75 @1586803708.138
16069036.66 @1586803723.138
16069049.08 @1586803738.138
16069061.58 @1586803753.138
16069074.35 @1586803768.138
16069084.85 @1586803783.138
16069097.27 @1586803798.138
16069109.04 @1586803813.138
16069120.5 @1586803828.138
16069133.05 @1586803843.138
16069145.18 @1586803858.138
16069156.91 @1586803873.137
16069168.14 @1586803888.138
16069180.44 @1586803903.138
16069192.16 @1586803918.138
16069202.6 @1586803933.138
16069214.49 @1586803948.137
16069226.49 @1586803963.137
16069239.11 @1586803978.138
16069249.84 @1586803993.138
16069262.32 @1586804008.138
16069274.87 @1586804023.138
16069285.84 @1586804038.138
16069297.06 @1586804053.137
16069309.65 @1586804068.138
16069322.31 @1586804083.138
16069333.94 @1586804098.138
16069345.95 @1586804113.138
16069358.37 @1586804128.137
16069370.37 @1586804143.138
16069382.78 @1586804158.138
16069394.29 @1586804173.138
16069405.93 @1586804188.138
16069416.98 @1586804203.138
16069428.71 @1586804218.137
16069441.23 @1586804233.137
16069452.14 @1586804248.138
16069465.02 @1586804263.138
16069477.65 @1586804278.138
16069489.47 @1586804293.138
16069500.15 @1586804308.138
16069512.4 @1586804323.138
16069523.89 @1586804338.138
16069535.21 @1586804353.138
16069542.89 @1586804368.138
16069554.53 @1586804383.137
16069566.29 @1586804398.138
16069577.35 @1586804413.138
16069589.57 @1586804428.138
16069601.32 @1586804443.138
16069613.37 @1586804458.138
16069623.19 @1586804473.137
16069634.87 @1586804488.138
16069647.06 @1586804503.138
16069658.97 @1586804518.137
16069670.52 @1586804533.137
16069682.75 @1586804548.138
16069694.29 @1586804563.138
16069704.78 @1586804578.138
16069717.09 @1586804593.138
16069729.26 @1586804608.138
16069741.15 @1586804623.138
16069753.06 @1586804638.137
16069765.1 @1586804653.138
16069777.46 @1586804668.138
16069788.77 @1586804683.137
16069801.5 @1586804698.138
16069812.81 @1586804713.138
16069823.86 @1586804728.137
16069836.79 @1586804743.138
16069848.87 @1586804758.138
16069860.99 @1586804773.138
16069872.89 @1586804788.138
16069884.82 @1586804803.137
16069897.47 @1586804818.138
16069908.74 @1586804833.138
16069920.43 @1586804848.138
16069932.42 @1586804863.138
16069944.56 @1586804878.138
16069956.09 @1586804893.138
16069968.58 @1586804908.138
16069980.59 @1586804923.137
16069993.05 @1586804938.137
16070005.68 @1586804953.138
16070017.97 @1586804968.138
16070030.59 @1586804983.138
16070041.88 @1586804998.138
16070053.54 @1586805013.137
node_cpu_seconds_total{cpu="0",instance="XXX:9100",mode="idle"}
3429654.02 @1586801427.17
3429668.74 @1586801442.17
3429683.52 @1586801457.17
3429698.23 @1586801472.17
3429713 @1586801487.17
3429727.72 @1586801502.17
3429742.5 @1586801517.17
3429757.22 @1586801532.17
3429772 @1586801547.17
3429786.73 @1586801562.17
3430268.84 @1586802057.17
3430283.58 @1586802072.17
3430298.35 @1586802087.17
3430313.04 @1586802102.17
3430327.82 @1586802117.17
3430342.52 @1586802132.17
3430357.29 @1586802147.17
3430372.01 @1586802162.17
3430386.79 @1586802177.17
3430401.5 @1586802192.17
3430416.28 @1586802207.17
3430430.99 @1586802222.17
3430445.78 @1586802237.17
3430460.5 @1586802252.17
3430475.27 @1586802267.17
3430490 @1586802282.17
3430504.78 @1586802297.17
3430519.53 @1586802312.17
3430534.3 @1586802327.17
3430549.03 @1586802342.17
3430563.82 @1586802357.17
3430578.54 @1586802372.17
3430593.32 @1586802387.17
3430608.05 @1586802402.17
3430622.8 @1586802417.17
3430637.53 @1586802432.17
3430652.29 @1586802447.17
3430667.02 @1586802462.17
3430681.79 @1586802477.17
3430696.5 @1586802492.17
3430711.25 @1586802507.17
3430725.98 @1586802522.17
3430740.77 @1586802537.17
3430755.5 @1586802552.17
3430770.29 @1586802567.17
3430785.01 @1586802582.17
3430799.78 @1586802597.17
3430813.48 @1586802612.17
3430828.22 @1586802627.17
3430842.96 @1586802642.172
3430857.74 @1586802657.17
3430872.44 @1586802672.17
3430887.23 @1586802687.17
3430901.97 @1586802702.17
3430916.75 @1586802717.17
3430931.47 @1586802732.17
3430946.23 @1586802747.17
3430960.96 @1586802762.17
3430975.73 @1586802777.17
3430990.44 @1586802792.17
3431005.2 @1586802807.17
3431016.7 @1586802822.177
3431030.78 @1586802837.17
3431045.46 @1586802852.17
3431060.22 @1586802867.17
3431074.97 @1586802882.17
3431089.74 @1586802897.17
3431104.44 @1586802912.17
3431119.2 @1586802927.17
3431133.93 @1586802942.17
3431148.7 @1586802957.17
3431163.42 @1586802972.17
3431178.17 @1586802987.17
3431192.9 @1586803002.17
3431207.66 @1586803017.17
3431222.36 @1586803032.17
3431237.13 @1586803047.17
3431251.87 @1586803062.17
3431266.66 @1586803077.17
3431281.38 @1586803092.17
3431296.17 @1586803107.17
3431310.91 @1586803122.17
3431325.68 @1586803137.17
3431340.41 @1586803152.17
3431355.19 @1586803167.17
3431369.92 @1586803182.17
3431384.69 @1586803197.17
3431399.36 @1586803212.17
3431414.13 @1586803227.17
3431428.86 @1586803242.17
3431443.61 @1586803257.17
3431458.33 @1586803272.17
3431473.1 @1586803287.17
3431487.81 @1586803302.17
3431502.58 @1586803317.17
3431517.26 @1586803332.17
3431532.03 @1586803347.17
3431546.76 @1586803362.17
3431561.53 @1586803377.17
3431576.26 @1586803392.17
3431591.02 @1586803407.17
3431605.76 @1586803422.17
3431620.52 @1586803437.17
3431635.24 @1586803452.17
3431650.02 @1586803467.17
3431664.76 @1586803482.17
3431679.53 @1586803497.17
3431694.26 @1586803512.17
3431709.05 @1586803527.17
3431723.8 @1586803542.17
3431738.57 @1586803557.17
3431753.28 @1586803572.17
3431768.07 @1586803587.17
3431782.8 @1586803602.17
3431797.57 @1586803617.17
3431812.28 @1586803632.17
3431827.06 @1586803647.17
3431841.79 @1586803662.172
3431856.56 @1586803677.17
3431871.28 @1586803692.17
3431886.03 @1586803707.17
3431900.75 @1586803722.17
3431915.52 @1586803737.17
3431930.24 @1586803752.17
3431944.99 @1586803767.17
3431959.72 @1586803782.17
3431974.47 @1586803797.17
3431987.99 @1586803812.17
3431999.3 @1586803827.17
3432014.03 @1586803842.17
3432028.76 @1586803857.17
3432043.46 @1586803872.17
3432058.23 @1586803887.17
3432072.98 @1586803902.17
3432087.77 @1586803917.17
3432102.47 @1586803932.17
3432117.25 @1586803947.17
3432131.99 @1586803962.17
3432146.77 @1586803977.17
3432161.5 @1586803992.17
3432176.26 @1586804007.17
3432191.01 @1586804022.176
3432205.78 @1586804037.17
3432220.51 @1586804052.17
3432235.29 @1586804067.17
3432250.04 @1586804082.17
3432264.8 @1586804097.17
3432279.52 @1586804112.17
3432294.28 @1586804127.17
3432309.02 @1586804142.17
3432323.78 @1586804157.17
3432338.49 @1586804172.17
3432353.26 @1586804187.17
3432367.99 @1586804202.17
3432382.76 @1586804217.17
3432397.49 @1586804232.17
3432412.25 @1586804247.17
3432426.99 @1586804262.17
3432441.77 @1586804277.17
3432456.48 @1586804292.17
3432471.25 @1586804307.17
3432485.97 @1586804322.17
3432500.73 @1586804337.17
3432515.43 @1586804352.17
3432530.2 @1586804367.17
3432544.93 @1586804382.17
3432559.7 @1586804397.17
3432574.42 @1586804412.17
3432589.2 @1586804427.17
3432603.93 @1586804442.17
3432618.7 @1586804457.17
3432633.31 @1586804472.17
3432648.09 @1586804487.17
3432662.82 @1586804502.17
3432677.58 @1586804517.17
3432692.28 @1586804532.17
3432707.05 @1586804547.17
3432721.78 @1586804562.17
3432736.55 @1586804577.17
3432751.26 @1586804592.17
3432766.05 @1586804607.17
3432780.76 @1586804622.17
3432795.53 @1586804637.17
3432810.23 @1586804652.17
3432825.01 @1586804667.17
3432839.75 @1586804682.17
3432854.52 @1586804697.17
3432869.25 @1586804712.17
3432884.02 @1586804727.17
3432898.75 @1586804742.17
3432913.52 @1586804757.17
3432925.73 @1586804772.17
3432937.82 @1586804787.17
3432952.56 @1586804802.17
3432967.33 @1586804817.17
3432982.04 @1586804832.17
3432996.8 @1586804847.17
3433011.54 @1586804862.17
3433026.31 @1586804877.17
3433041.03 @1586804892.17
3433055.78 @1586804907.17
3433070.51 @1586804922.17
3433085.28 @1586804937.17
3433100.01 @1586804952.17
3433114.77 @1586804967.17
3433129.47 @1586804982.171
3433144.24 @1586804997.17
3433158.94 @1586805012.17
node_cpu_seconds_total{cpu="1",instance="XXX:9100",mode="idle"}
3427002.53 @1586801427.17
3427017.24 @1586801442.17
3427032 @1586801457.17
3427046.72 @1586801472.17
3427061.49 @1586801487.17
3427076.21 @1586801502.17
3427090.99 @1586801517.17
3427105.7 @1586801532.17
3427120.47 @1586801547.17
3427135.19 @1586801562.17
3427616.55 @1586802057.17
3427631.26 @1586802072.17
3427646.01 @1586802087.17
3427660.7 @1586802102.17
3427675.46 @1586802117.17
3427690.14 @1586802132.17
3427704.89 @1586802147.17
3427719.6 @1586802162.17
3427734.37 @1586802177.17
3427749.1 @1586802192.17
3427763.86 @1586802207.17
3427778.58 @1586802222.17
3427793.36 @1586802237.17
3427808.09 @1586802252.17
3427822.87 @1586802267.17
3427837.61 @1586802282.17
3427852.39 @1586802297.17
3427867.11 @1586802312.17
3427881.88 @1586802327.17
3427896.6 @1586802342.17
3427911.35 @1586802357.17
3427926.07 @1586802372.17
3427940.84 @1586802387.17
3427955.57 @1586802402.17
3427970.33 @1586802417.17
3427985.04 @1586802432.17
3427999.8 @1586802447.17
3428014.53 @1586802462.17
3428029.29 @1586802477.17
3428044.01 @1586802492.17
3428058.77 @1586802507.17
3428073.48 @1586802522.17
3428088.23 @1586802537.17
3428102.94 @1586802552.17
3428117.69 @1586802567.17
3428132.39 @1586802582.17
3428147.16 @1586802597.17
3428160.95 @1586802612.17
3428175.7 @1586802627.17
3428190.42 @1586802642.172
3428205.2 @1586802657.17
3428219.91 @1586802672.17
3428234.68 @1586802687.17
3428249.41 @1586802702.17
3428264.17 @1586802717.17
3428278.89 @1586802732.17
3428293.64 @1586802747.17
3428308.36 @1586802762.17
3428323.13 @1586802777.17
3428337.83 @1586802792.17
3428352.59 @1586802807.17
3428363.97 @1586802822.177
3428376.47 @1586802837.17
3428391.11 @1586802852.17
3428405.87 @1586802867.17
3428420.59 @1586802882.17
3428435.34 @1586802897.17
3428450.04 @1586802912.17
3428464.8 @1586802927.17
3428479.53 @1586802942.17
3428494.3 @1586802957.17
3428508.99 @1586802972.17
3428523.75 @1586802987.17
3428538.47 @1586803002.17
3428553.21 @1586803017.17
3428567.93 @1586803032.17
3428582.7 @1586803047.17
3428597.42 @1586803062.17
3428612.18 @1586803077.17
3428626.9 @1586803092.17
3428641.68 @1586803107.17
3428656.39 @1586803122.17
3428671.17 @1586803137.17
3428685.89 @1586803152.17
3428700.65 @1586803167.17
3428715.39 @1586803182.17
3428730.16 @1586803197.17
3428744.84 @1586803212.17
3428759.61 @1586803227.17
3428774.31 @1586803242.17
3428789.06 @1586803257.17
3428803.78 @1586803272.17
3428818.51 @1586803287.17
3428833.24 @1586803302.17
3428847.98 @1586803317.17
3428862.65 @1586803332.17
3428877.42 @1586803347.17
3428892.14 @1586803362.17
3428906.9 @1586803377.17
3428921.62 @1586803392.17
3428936.39 @1586803407.17
3428951.13 @1586803422.17
3428965.88 @1586803437.17
3428980.59 @1586803452.17
3428995.37 @1586803467.17
3429010.1 @1586803482.17
3429024.87 @1586803497.17
3429039.58 @1586803512.17
3429054.34 @1586803527.17
3429069.1 @1586803542.17
3429083.85 @1586803557.17
3429098.56 @1586803572.17
3429113.33 @1586803587.17
3429128.05 @1586803602.17
3429142.83 @1586803617.17
3429157.54 @1586803632.17
3429172.31 @1586803647.17
3429187.06 @1586803662.172
3429201.83 @1586803677.17
3429216.55 @1586803692.17
3429231.32 @1586803707.17
3429246.05 @1586803722.17
3429260.81 @1586803737.17
3429275.55 @1586803752.17
3429290.3 @1586803767.17
3429305.01 @1586803782.17
3429319.78 @1586803797.17
3429333.57 @1586803812.17
3429344.41 @1586803827.17
3429359.13 @1586803842.17
3429373.88 @1586803857.17
3429388.58 @1586803872.17
3429403.34 @1586803887.17
3429418.08 @1586803902.17
3429432.83 @1586803917.17
3429447.52 @1586803932.17
3429462.31 @1586803947.17
3429477.03 @1586803962.17
3429491.8 @1586803977.17
3429506.5 @1586803992.17
3429521.27 @1586804007.17
3429536.02 @1586804022.176
3429550.77 @1586804037.17
3429565.49 @1586804052.17
3429580.26 @1586804067.17
3429594.97 @1586804082.17
3429609.74 @1586804097.17
3429624.43 @1586804112.17
3429639.17 @1586804127.17
3429653.9 @1586804142.17
3429668.63 @1586804157.17
3429683.35 @1586804172.17
3429698.1 @1586804187.17
3429712.82 @1586804202.17
3429727.58 @1586804217.17
3429742.28 @1586804232.17
3429757.03 @1586804247.17
3429771.76 @1586804262.17
3429786.53 @1586804277.17
3429801.23 @1586804292.17
3429815.99 @1586804307.17
3429830.67 @1586804322.17
3429845.46 @1586804337.17
3429860.13 @1586804352.17
3429874.91 @1586804367.17
3429889.63 @1586804382.17
3429904.41 @1586804397.17
3429919.07 @1586804412.17
3429933.82 @1586804427.17
3429948.53 @1586804442.17
3429963.3 @1586804457.17
3429977.93 @1586804472.17
3429992.69 @1586804487.17
3430007.4 @1586804502.17
3430022.17 @1586804517.17
3430036.86 @1586804532.17
3430051.62 @1586804547.17
3430066.34 @1586804562.17
3430081.1 @1586804577.17
3430095.79 @1586804592.17
3430110.55 @1586804607.17
3430125.29 @1586804622.17
3430140.05 @1586804637.17
3430154.72 @1586804652.17
3430169.47 @1586804667.17
3430184.2 @1586804682.17
3430198.96 @1586804697.17
3430213.69 @1586804712.17
3430228.46 @1586804727.17
3430243.21 @1586804742.17
3430257.99 @1586804757.17
3430270.51 @1586804772.17
3430283.09 @1586804787.17
3430297.81 @1586804802.17
3430312.57 @1586804817.17
3430327.29 @1586804832.17
3430342.05 @1586804847.17
3430356.79 @1586804862.17
3430371.55 @1586804877.17
3430386.27 @1586804892.17
3430401.02 @1586804907.17
3430415.77 @1586804922.17
3430430.54 @1586804937.17
3430445.28 @1586804952.17
3430460.04 @1586804967.17
3430474.73 @1586804982.171
3430489.5 @1586804997.17
3430504.18 @1586805012.17
