Rafit, I would recommend using shard durations that are as long as reasonable. I see a lot of 1 hour shards, which will lead to significant RAM usage. Each shard queried has a fixed RAM overhead, per query.
On Tue, Oct 4, 2016 at 12:21 PM, Rafit Izhak Ratzin < [email protected]> wrote: > - Please see the attached continuous-queries, shards, and stats files. > - No regular expression > - Measurement names length are up to 26chars. > - Tag/fields name are about 10chars plus/minus, > - Tag/fields values might be up to 20 char (if string) > - Pushing every 5 seconds or up to 8192 points (which satisfies first) > - Using the java api "InfluxDB", function "write", to write "BatchPoints" > > > Thanks for your help, > --Rafit > > What are the actual CQs being run? > Queries aren't of equal cost, so perhaps the CQs are particularly > punishing. > What's the user-initiated query load? > Do the queries involve regular expressions? Multiple AND/OR conditions, > especially on fields? Are they well bounded in time? > Are you writing points in the distant past or in the future? > Are you writing points out of chronological order? > Can you share the output from SHOW SHARDS? > Are you using long strings as measurement names, tag keys or values, or > field keys? > > On Tue, Sep 27, 2016 at 3:12 PM, Sean Beckett <[email protected]> wrote: > >> There's no reason for an otherwise average system to use 13GB of RAM with >> that CQ load and series cardinality. There must be something non-standard >> about your use case. >> >> What are the actual CQs being run? Queries aren't of equal cost, so >> perhaps the CQs are particularly punishing. >> What's the user-initiated query load? >> Do the queries involve regular expressions? Multiple AND/OR conditions, >> especially on fields? Are they well bounded in time? >> Are you writing points in the distant past or in the future? >> Are you writing points out of chronological order? >> Can you share the output from SHOW SHARDS? >> Are you using long strings as measurement names, tag keys or values, or >> field keys? >> >> >> On Tue, Sep 27, 2016 at 12:03 PM, Rafit Izhak Ratzin < >> [email protected]> wrote: >> >>> Hi Sean, >>> >>> I reduced the number of CQ to 18, >>> 6 sample every 1m, >>> 6 sample every 1h >>> 6 sample every 6h >>> >>> number of series 40889, >>> >>> I upgrade to influx 1.0 >>> Still expressing memory issues, after few days influxdb will use the >>> entire available memory (13G) and will crash. >>> >>> Thanks, >>> --Rafit >>> >>> >>> >>> On Mon, Sep 12, 2016 at 12:00 PM, Sean Beckett <[email protected]> >>> wrote: >>> >>>> 27 CQs could be responsible for high RAM usage if they are complex or >>>> frequent queries. What is the output from `SHOW CONTINUOUS QUERIES`? >>>> >>>> With 1.0 GA released, I encourage you to upgrade. That might address >>>> some RAM issues. >>>> >>>> On Thu, Sep 8, 2016 at 12:06 AM, Rafit Izhak Ratzin < >>>> [email protected]> wrote: >>>> >>>>> Hi Sean, >>>>> >>>>> The current verified number of series is : 41712 >>>>> >>>>> 27 CQ, >>>>> >>>>> I am using enableBatch: >>>>> influxClient.enableBatch(10000, 3000, TimeUnit.MILLISECONDS); >>>>> >>>>> at least 8192 calls of: >>>>> batch_points.point(metric_point); >>>>> >>>>> before calling write: >>>>> influxClient.write(batch_points); >>>>> >>>>> >>>>> Please let me know if you have more questions (more stats are >>>>> available in the end of this email), >>>>> I appreciate your help, >>>>> Thank you! >>>>> --Rafit >>>>> >>>>> some more stats: >>>>> >>>>> select * from runtime where time > now() - 10s >>>>> name: runtime >>>>> ------------- >>>>> time Alloc Frees >>>>> HeapAlloc HeapIdle HeapInUse HeapObjects >>>>> HeapReleased HeapSys Lookups Mallocs NumGC >>>>> NumGoroutine PauseTotalNs Sys TotalAlloc >>>>> clusterID hostname nodeID >>>>> 1473312983000000000 153833696 345337583 >>>>> 153833696 29990912 204431360 2751982 >>>>> 0 234422272 491 348089565 246 >>>>> 134 122381212 248941072 15169333136 >>>>> 2765793636717751283 ??????.?????????????.??? 0 >>>>> >>>>> select * from "database" where time > now() - 10s >>>>> name: database >>>>> -------------- >>>>> time clusterID database >>>>> hostname nodeID numMeasurements numSeries >>>>> 1473313073000000000 2765793636717751283 _internal >>>>> ??????.????????????.??? 0 11 131 >>>>> 1473313073000000000 2765793636717751283 analytics >>>>> ??????.????????????.??? 0 25 41581 >>>>> >>>>> select * from "shard" where time > now() - 10s >>>>> name: shard >>>>> ----------- >>>>> time clusterID database >>>>> engine fieldsCreate hostname id >>>>> nodeID path retentionPolicy seriesCreate >>>>> writePointsOk writeReq >>>>> 1473313223000000000 2765793636717751283 _internal >>>>> tsm1 0 ??????.????????????.??? 31 >>>>> 0 /var/lib/influxdb/data/_internal/monitor/31 >>>>> monitor 9391 122 >>>>> >>>>> select * from "write" where time > now() - 10s >>>>> name: write >>>>> ----------- >>>>> time clusterID >>>>> hostname nodeID pointReq >>>>> pointReqLocal req subWriteDrop subWriteOk writeOk >>>>> 1473313293000000000 2765793636717751283 >>>>> ??????.????????????.??? 0 9930 >>>>> 9930 129 129 129 >>>>> >>>>> show stats >>>>> name: cq >>>>> -------- >>>>> queryOk >>>>> 36342 >>>>> >>>>> >>>>> name: database >>>>> tags: database=_internal >>>>> numMeasurements numSeries >>>>> --------------- --------- >>>>> 11 131 >>>>> >>>>> >>>>> name: database >>>>> tags: database=analytics >>>>> numMeasurements numSeries >>>>> --------------- --------- >>>>> 25 41581 >>>>> >>>>> >>>>> name: httpd >>>>> tags: bind=:8086 >>>>> pingReq queryReq queryReqDurationNs queryRespBytes >>>>> req reqActive reqDurationNs >>>>> ------- -------- ------------------ -------------- >>>>> --- --------- ------------- >>>>> 1 51 764949154 2509363 >>>>> 52 1 773122523 >>>>> >>>>> >>>>> name: queryExecutor >>>>> ------------------- >>>>> queriesActive queryDurationNs >>>>> 1 1677180844 >>>>> >>>>> >>>>> name: shard >>>>> tags: database=_internal, engine=tsm1, id=31, >>>>> path=/var/lib/influxdb/data/_internal/monitor/31, >>>>> retentionPolicy=monitor >>>>> fieldsCreate writePointsOk writeReq >>>>> ------------ ------------- -------- >>>>> 0 10623 138 >>>>> >>>>> >>>>> name: subscriber >>>>> ---------------- >>>>> pointsWritten >>>>> 10623 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=_internal, path=/var/lib/influxdb/data/_internal/monitor/1, >>>>> retentionPolicy=monitor >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1427909 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=_internal, >>>>> path=/var/lib/influxdb/data/_internal/monitor/10, >>>>> retentionPolicy=monitor >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1427908 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=_internal, >>>>> path=/var/lib/influxdb/data/_internal/monitor/31, >>>>> retentionPolicy=monitor >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1428182 0 >>>>> 0 9594304 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, path=/var/lib/influxdb/data/analytics/default/2, >>>>> retentionPolicy=default >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1427908 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/17, >>>>> retentionPolicy=one_day >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1427909 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/18, >>>>> retentionPolicy=one_day >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1404711 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/19, >>>>> retentionPolicy=one_day >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1389395 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/20, >>>>> retentionPolicy=one_day >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1405509 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/21, >>>>> retentionPolicy=one_day >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1397407 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/22, >>>>> retentionPolicy=one_day >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1396496 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/23, >>>>> retentionPolicy=one_day >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1396236 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/24, >>>>> retentionPolicy=one_day >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1403847 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/25, >>>>> retentionPolicy=one_day >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1414572 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/26, >>>>> retentionPolicy=one_day >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1414691 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/27, >>>>> retentionPolicy=one_day >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1405726 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/28, >>>>> retentionPolicy=one_day >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1395702 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/29, >>>>> retentionPolicy=one_day >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1416754 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/30, >>>>> retentionPolicy=one_day >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1419778 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_month/13, >>>>> retentionPolicy=one_month >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1427868 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_month/5, >>>>> retentionPolicy=one_month >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1423398 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_week/12, >>>>> retentionPolicy=one_week >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1422570 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_week/4, >>>>> retentionPolicy=one_week >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1420522 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_cache >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_year/6, >>>>> retentionPolicy=one_year >>>>> WALCompactionTimeMs cacheAgeMs cachedBytes >>>>> diskBytes memBytes snapshotCount >>>>> ------------------- ---------- ----------- >>>>> --------- -------- ------------- >>>>> 0 1426600 0 >>>>> 0 0 0 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=_internal, path=/var/lib/influxdb/data/_internal/monitor/1, >>>>> retentionPolicy=monitor >>>>> diskBytes >>>>> --------- >>>>> 330422 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=_internal, >>>>> path=/var/lib/influxdb/data/_internal/monitor/10, >>>>> retentionPolicy=monitor >>>>> diskBytes >>>>> --------- >>>>> 2321520 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, path=/var/lib/influxdb/data/analytics/default/2, >>>>> retentionPolicy=default >>>>> diskBytes >>>>> --------- >>>>> 22524306 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/17, >>>>> retentionPolicy=one_day >>>>> diskBytes >>>>> --------- >>>>> 243080989 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/18, >>>>> retentionPolicy=one_day >>>>> diskBytes >>>>> --------- >>>>> 239262941 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/19, >>>>> retentionPolicy=one_day >>>>> diskBytes >>>>> --------- >>>>> 248413477 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/20, >>>>> retentionPolicy=one_day >>>>> diskBytes >>>>> --------- >>>>> 244294838 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/21, >>>>> retentionPolicy=one_day >>>>> diskBytes >>>>> --------- >>>>> 239914567 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/22, >>>>> retentionPolicy=one_day >>>>> diskBytes >>>>> --------- >>>>> 252412434 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/23, >>>>> retentionPolicy=one_day >>>>> diskBytes >>>>> --------- >>>>> 245078392 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/24, >>>>> retentionPolicy=one_day >>>>> diskBytes >>>>> --------- >>>>> 239291222 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/25, >>>>> retentionPolicy=one_day >>>>> diskBytes >>>>> --------- >>>>> 250779604 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/26, >>>>> retentionPolicy=one_day >>>>> diskBytes >>>>> --------- >>>>> 243886546 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/27, >>>>> retentionPolicy=one_day >>>>> diskBytes >>>>> --------- >>>>> 238989758 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/28, >>>>> retentionPolicy=one_day >>>>> diskBytes >>>>> --------- >>>>> 250148686 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/29, >>>>> retentionPolicy=one_day >>>>> diskBytes >>>>> --------- >>>>> 174987063 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_day/30, >>>>> retentionPolicy=one_day >>>>> diskBytes >>>>> --------- >>>>> 68603390 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_month/13, >>>>> retentionPolicy=one_month >>>>> diskBytes >>>>> --------- >>>>> 31862665 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_month/5, >>>>> retentionPolicy=one_month >>>>> diskBytes >>>>> --------- >>>>> 29406087 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_week/12, >>>>> retentionPolicy=one_week >>>>> diskBytes >>>>> --------- >>>>> 90157429 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_week/4, >>>>> retentionPolicy=one_week >>>>> diskBytes >>>>> --------- >>>>> 144493656 >>>>> >>>>> >>>>> name: tsm1_filestore >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/data/analytics/one_year/6, >>>>> retentionPolicy=one_year >>>>> diskBytes >>>>> --------- >>>>> 49154983 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=_internal, path=/var/lib/influxdb/wal/_internal/monitor/1, >>>>> retentionPolicy=monitor >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=_internal, path=/var/lib/influxdb/wal/_internal/monitor/10, >>>>> retentionPolicy=monitor >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=_internal, path=/var/lib/influxdb/wal/_internal/monitor/31, >>>>> retentionPolicy=monitor >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 725550 11936787 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/default/2, >>>>> retentionPolicy=default >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_day/17, >>>>> retentionPolicy=one_day >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_day/18, >>>>> retentionPolicy=one_day >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_day/19, >>>>> retentionPolicy=one_day >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_day/20, >>>>> retentionPolicy=one_day >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_day/21, >>>>> retentionPolicy=one_day >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_day/22, >>>>> retentionPolicy=one_day >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_day/23, >>>>> retentionPolicy=one_day >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_day/24, >>>>> retentionPolicy=one_day >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_day/25, >>>>> retentionPolicy=one_day >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_day/26, >>>>> retentionPolicy=one_day >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_day/27, >>>>> retentionPolicy=one_day >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_day/28, >>>>> retentionPolicy=one_day >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_day/29, >>>>> retentionPolicy=one_day >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_day/30, >>>>> retentionPolicy=one_day >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/wal/analytics/one_month/13, >>>>> retentionPolicy=one_month >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/wal/analytics/one_month/5, >>>>> retentionPolicy=one_month >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, >>>>> path=/var/lib/influxdb/wal/analytics/one_week/12, >>>>> retentionPolicy=one_week >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_week/4, >>>>> retentionPolicy=one_week >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: tsm1_wal >>>>> tags: database=analytics, path=/var/lib/influxdb/wal/analytics/one_year/6, >>>>> retentionPolicy=one_year >>>>> currentSegmentDiskBytes oldSegmentsDiskBytes >>>>> ----------------------- -------------------- >>>>> 0 0 >>>>> >>>>> >>>>> name: write >>>>> ----------- >>>>> pointReq pointReqLocal req subWriteOk writeOk >>>>> 10623 10623 138 138 138 >>>>> >>>>> >>>>> name: runtime >>>>> ------------- >>>>> Alloc Frees HeapAlloc HeapIdle >>>>> HeapInUse HeapObjects HeapReleased HeapSys Lookups >>>>> Mallocs NumGC NumGoroutine PauseTotalNs Sys >>>>> TotalAlloc >>>>> 175354504 384279146 175354504 18333696 >>>>> 221331456 3414371 0 239665152 518 >>>>> 387693517 263 139 132846894 254351888 16680835416 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Sep 7, 2016 at 8:36 PM, Sean Beckett <[email protected]> >>>>> wrote: >>>>> >>>>>> 45k series should not consume 64GB of RAM. Are there any CQs running? >>>>>> How many writes per second? Are they batched? What's the user-initiated >>>>>> query load? >>>>>> >>>>>> Can you verify the series count using >>>>>> >>>>>> SELECT sum(numSeries) AS “total_series" FROM “_internal".."database" >>>>>> WHERE time > now() - 10s >>>>>> >>>>>> On Wed, Sep 7, 2016 at 6:07 PM, <[email protected]> wrote: >>>>>> >>>>>>> Hi Sean, >>>>>>> I am running into a similar problem, >>>>>>> RAM size is 64G >>>>>>> number of series is 45,158 >>>>>>> influxdb uses all the memory and crashes, >>>>>>> Once it crashes, after restarting is will fill up the memory within >>>>>>> 10 minutes or so and will crash again. >>>>>>> I am using the java api to push data, >>>>>>> >>>>>>> Thanks, >>>>>>> --Rafit >>>>>>> >>>>>>> -- >>>>>>> Remember to include the InfluxDB version number with all issue >>>>>>> reports >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "InfluxDB" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/influxdb/6d3e06d7-c36c-4da >>>>>>> 1-908c-9b0407646e31%40googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Sean Beckett >>>>>> Director of Support and Professional Services >>>>>> InfluxDB >>>>>> >>>>>> -- >>>>>> Remember to include the InfluxDB version number with all issue reports >>>>>> --- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "InfluxDB" group. >>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>>>> pic/influxdb/fiQzEy1aWP8/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/influxdb/CALGqCvNN10UeGLDq >>>>>> 00juNdjrRD-qpRD219bFtKzJPYHXDxg%3D9Q%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/influxdb/CALGqCvNN10UeGLDq00juNdjrRD-qpRD219bFtKzJPYHXDxg%3D9Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Rafit Izhak Ratzin, PhD >>>>> Computer Science Department >>>>> UCLA >>>>> >>>>> -- >>>>> Remember to include the InfluxDB version number with all issue reports >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "InfluxDB" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/influxdb/CAFAManek-igJM61g >>>>> _2Qz%3Dhf1q1Ccro91-ZFEecPX_j5zUkY_4Q%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/influxdb/CAFAManek-igJM61g_2Qz%3Dhf1q1Ccro91-ZFEecPX_j5zUkY_4Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Sean Beckett >>>> Director of Support and Professional Services >>>> InfluxDB >>>> >>>> -- >>>> Remember to include the InfluxDB version number with all issue reports >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "InfluxDB" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>> pic/influxdb/fiQzEy1aWP8/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/influxdb. >>>> To view this discussion on the web visit https://groups.google.com/d/ms >>>> gid/influxdb/CALGqCvOkTya7MiVzMeoQhMrPtAQZ%2BhNrbO6%3DkgC8-J >>>> EooZuAgQ%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/influxdb/CALGqCvOkTya7MiVzMeoQhMrPtAQZ%2BhNrbO6%3DkgC8-JEooZuAgQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Rafit Izhak Ratzin, PhD >>> Computer Science Department >>> UCLA >>> >>> -- >>> Remember to include the InfluxDB version number with all issue reports >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "InfluxDB" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/influxdb. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/influxdb/CAFAManc%3D-oFHGtXDJy_4OLQNY%2B1xrDqk%3DRsHAjYe >>> P4tyCx%3DVkw%40mail.gmail.com >>> <https://groups.google.com/d/msgid/influxdb/CAFAManc%3D-oFHGtXDJy_4OLQNY%2B1xrDqk%3DRsHAjYeP4tyCx%3DVkw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Sean Beckett >> Director of Support and Professional Services >> InfluxDB >> >> -- >> Remember to include the InfluxDB version number with all issue reports >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "InfluxDB" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/to >> pic/influxdb/fiQzEy1aWP8/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/influxdb. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/influxdb/CALGqCvMCftO5W4ouhxQBg9M%2BT%2BGBfiSuCFq7qHcnmF >> qtGfDeNw%40mail.gmail.com >> <https://groups.google.com/d/msgid/influxdb/CALGqCvMCftO5W4ouhxQBg9M%2BT%2BGBfiSuCFq7qHcnmFqtGfDeNw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Rafit Izhak Ratzin, PhD > Computer Science Department > UCLA > > -- > Remember to include the InfluxDB version number with all issue reports > --- > You received this message because you are subscribed to the Google Groups > "InfluxDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/influxdb. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/influxdb/CAFAMand8Vz9AVOxmGwR0ivnSskmzBQq6iRtmFuHPuuYj3FAA5w%40mail. > gmail.com > <https://groups.google.com/d/msgid/influxdb/CAFAMand8Vz9AVOxmGwR0ivnSskmzBQq6iRtmFuHPuuYj3FAA5w%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Sean Beckett Director of Support and Professional Services InfluxDB -- Remember to include the version number! --- You received this message because you are subscribed to the Google Groups "InfluxData" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CALGqCvP_ZZvyaWhf9R%2Bg-p%2BZfjLs6SM0p6ES4Xakt6yxKqG4sQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
