When the system shuts down the WAL may have points still in it. On restart, the system must first flush the WAL before receiving new points. The idea being if the node already has pending WAL writes, there's no reason to allow new incoming writes until the WAL is empty.
On Mon, Aug 1, 2016 at 9:58 AM, Jan Kis <[email protected]> wrote: > To gain a bit better understanding of influx: what is influx writing into > the wal files when it is restarted and not receiving any requests yet? I > thought that the wal files are populated with data coming into influx > before they are stored in the data directory. > > > On Monday, August 1, 2016 at 5:50:07 PM UTC+2, Jan Kis wrote: >> >> Removing the latest shard from >> directory /data/influxdb/data/mydb/default/15 solved the issue but I am >> still interested in whether there is a bug or whether we have done >> something stupid. >> >> On Monday, August 1, 2016 at 1:25:49 PM UTC+2, Jan Kis wrote: >>> >>> I tried to move away the whole wal directory. That did not help I still >>> see the same exception. >>> >>> On Monday, August 1, 2016 at 12:58:31 PM UTC+2, Jan Kis wrote: >>>> >>>> Hi Guys, >>>> >>>> today we experienced the "panic: runtime error: slice bounds out of >>>> range" in influxdb 0.13. This happened while we were dropping some >>>> measurements. Currently, we can not get influx up and running. We see the >>>> same error on restart. I include a more detailed snippet from the logs >>>> below. >>>> >>>> Any help is appreciated, >>>> Thank you >>>> >>>> [tsm1] 2016/08/01 12:45:40 beginning level 2 compaction of group 0, 1 >>>> TSM files >>>> [tsm1] 2016/08/01 12:45:40 compacting level 2 group (0) >>>> /data/influxdb/data/mydb/default/13/000000164-000000002.tsm (#0) >>>> [shard] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/126 >>>> database index loaded in 66.51752ms >>>> [store] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/126 opened >>>> in 70.212921ms >>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL starting with 10485760 segment >>>> size >>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL writing to >>>> /data/influxdb/wal/mydb/default/134 >>>> [filestore] 2016/08/01 12:45:40 >>>> /data/influxdb/data/mydb/default/134/000000090-000000003.tsm (#0) opened in >>>> 1.669208ms >>>> [cacheloader] 2016/08/01 12:45:40 reading file >>>> /data/influxdb/wal/mydb/default/134/_00266.wal, size 144183 >>>> [cacheloader] 2016/08/01 12:45:40 reading file >>>> /data/influxdb/wal/mydb/default/134/_00296.wal, size 0 >>>> [shard] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/118 >>>> database index loaded in 76.444773ms >>>> [store] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/118 opened >>>> in 95.187655ms >>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL starting with 10485760 segment >>>> size >>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL writing to >>>> /data/influxdb/wal/mydb/default/14 >>>> [filestore] 2016/08/01 12:45:40 >>>> /data/influxdb/data/mydb/default/14/000000148-000000004.tsm (#1) opened in >>>> 67.574µs >>>> [filestore] 2016/08/01 12:45:40 >>>> /data/influxdb/data/mydb/default/14/000000147-000000004.tsm (#0) opened in >>>> 1.611877ms >>>> [cacheloader] 2016/08/01 12:45:40 reading file >>>> /data/influxdb/wal/mydb/default/14/_00302.wal, size 0 >>>> [shard] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/13 >>>> database index loaded in 66.423537ms >>>> [store] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/13 opened >>>> in 70.154698ms >>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL starting with 10485760 segment >>>> size >>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL writing to >>>> /data/influxdb/wal/mydb/default/142 >>>> [filestore] 2016/08/01 12:45:40 >>>> /data/influxdb/data/mydb/default/142/000000086-000000003.tsm (#0) opened in >>>> 2.154482ms >>>> [cacheloader] 2016/08/01 12:45:40 reading file >>>> /data/influxdb/wal/mydb/default/142/_00325.wal, size 116400 >>>> [cacheloader] 2016/08/01 12:45:40 reading file >>>> /data/influxdb/wal/mydb/default/142/_00340.wal, size 0 >>>> [shard] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/14 >>>> database index loaded in 66.356981ms >>>> [store] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/14 opened >>>> in 68.922787ms >>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL starting with 10485760 segment >>>> size >>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL writing to >>>> /data/influxdb/wal/mydb/default/15 >>>> panic: runtime error: slice bounds out of range >>>> >>>> goroutine 496 [running]: >>>> panic(0xb5b240, 0xc82000e040) >>>> /usr/local/go/src/runtime/panic.go:481 +0x3e6 >>>> >>>> github.com/influxdata/influxdb/tsdb/engine/tsm1.(*mmapAccessor).init(0xc8235f78c0, >>>> 0x0, 0x0, 0x0) >>>> /root/go/src/ >>>> github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:879 +0x5a8 >>>> >>>> github.com/influxdata/influxdb/tsdb/engine/tsm1.NewTSMReader(0xc825842030, >>>> 0xc80b8156de, 0x0, 0x0) >>>> /root/go/src/ >>>> github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:182 +0x204 >>>> >>>> github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Open.func1(0xc824c1c180, >>>> 0xc8236cb080, 0x2, 0xc825842030) >>>> /root/go/src/ >>>> github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:332 +0x77 >>>> created by >>>> github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Open >>>> /root/go/src/ >>>> github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:342 >>>> +0x749 >>>> >>> -- > Remember to include the InfluxDB version number with all issue reports > --- > You received this message because you are subscribed to the Google Groups > "InfluxDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/influxdb. > To view this discussion on the web visit > https://groups.google.com/d/msgid/influxdb/5f0edf0a-227b-436d-bc82-05332f53c191%40googlegroups.com > <https://groups.google.com/d/msgid/influxdb/5f0edf0a-227b-436d-bc82-05332f53c191%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Sean Beckett Director of Support and Professional Services InfluxDB -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CALGqCvPpK-hFxdO3Zdv1PMR6hM7n4kCoK4vMmgObnzdzm%3DKfHA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
