To clarify, the WAL is persisted to disk, but the points aren't yet organized into the permanent TSM structure. They are queryable and write-able on a restart, but they are not fully persisted until flushed to a TSM file.
On Mon, Aug 1, 2016 at 3:58 PM, Sean Beckett <[email protected]> wrote: > When the system shuts down the WAL may have points still in it. On > restart, the system must first flush the WAL before receiving new points. > The idea being if the node already has pending WAL writes, there's no > reason to allow new incoming writes until the WAL is empty. > > On Mon, Aug 1, 2016 at 9:58 AM, Jan Kis <[email protected]> wrote: > >> To gain a bit better understanding of influx: what is influx writing into >> the wal files when it is restarted and not receiving any requests yet? I >> thought that the wal files are populated with data coming into influx >> before they are stored in the data directory. >> >> >> On Monday, August 1, 2016 at 5:50:07 PM UTC+2, Jan Kis wrote: >>> >>> Removing the latest shard from >>> directory /data/influxdb/data/mydb/default/15 solved the issue but I am >>> still interested in whether there is a bug or whether we have done >>> something stupid. >>> >>> On Monday, August 1, 2016 at 1:25:49 PM UTC+2, Jan Kis wrote: >>>> >>>> I tried to move away the whole wal directory. That did not help I still >>>> see the same exception. >>>> >>>> On Monday, August 1, 2016 at 12:58:31 PM UTC+2, Jan Kis wrote: >>>>> >>>>> Hi Guys, >>>>> >>>>> today we experienced the "panic: runtime error: slice bounds out of >>>>> range" in influxdb 0.13. This happened while we were dropping some >>>>> measurements. Currently, we can not get influx up and running. We see the >>>>> same error on restart. I include a more detailed snippet from the logs >>>>> below. >>>>> >>>>> Any help is appreciated, >>>>> Thank you >>>>> >>>>> [tsm1] 2016/08/01 12:45:40 beginning level 2 compaction of group 0, 1 >>>>> TSM files >>>>> [tsm1] 2016/08/01 12:45:40 compacting level 2 group (0) >>>>> /data/influxdb/data/mydb/default/13/000000164-000000002.tsm (#0) >>>>> [shard] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/126 >>>>> database index loaded in 66.51752ms >>>>> [store] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/126 >>>>> opened in 70.212921ms >>>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL starting with 10485760 segment >>>>> size >>>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL writing to >>>>> /data/influxdb/wal/mydb/default/134 >>>>> [filestore] 2016/08/01 12:45:40 >>>>> /data/influxdb/data/mydb/default/134/000000090-000000003.tsm (#0) opened >>>>> in >>>>> 1.669208ms >>>>> [cacheloader] 2016/08/01 12:45:40 reading file >>>>> /data/influxdb/wal/mydb/default/134/_00266.wal, size 144183 >>>>> [cacheloader] 2016/08/01 12:45:40 reading file >>>>> /data/influxdb/wal/mydb/default/134/_00296.wal, size 0 >>>>> [shard] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/118 >>>>> database index loaded in 76.444773ms >>>>> [store] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/118 >>>>> opened in 95.187655ms >>>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL starting with 10485760 segment >>>>> size >>>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL writing to >>>>> /data/influxdb/wal/mydb/default/14 >>>>> [filestore] 2016/08/01 12:45:40 >>>>> /data/influxdb/data/mydb/default/14/000000148-000000004.tsm (#1) opened in >>>>> 67.574µs >>>>> [filestore] 2016/08/01 12:45:40 >>>>> /data/influxdb/data/mydb/default/14/000000147-000000004.tsm (#0) opened in >>>>> 1.611877ms >>>>> [cacheloader] 2016/08/01 12:45:40 reading file >>>>> /data/influxdb/wal/mydb/default/14/_00302.wal, size 0 >>>>> [shard] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/13 >>>>> database index loaded in 66.423537ms >>>>> [store] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/13 opened >>>>> in 70.154698ms >>>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL starting with 10485760 segment >>>>> size >>>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL writing to >>>>> /data/influxdb/wal/mydb/default/142 >>>>> [filestore] 2016/08/01 12:45:40 >>>>> /data/influxdb/data/mydb/default/142/000000086-000000003.tsm (#0) opened >>>>> in >>>>> 2.154482ms >>>>> [cacheloader] 2016/08/01 12:45:40 reading file >>>>> /data/influxdb/wal/mydb/default/142/_00325.wal, size 116400 >>>>> [cacheloader] 2016/08/01 12:45:40 reading file >>>>> /data/influxdb/wal/mydb/default/142/_00340.wal, size 0 >>>>> [shard] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/14 >>>>> database index loaded in 66.356981ms >>>>> [store] 2016/08/01 12:45:40 /data/influxdb/data/mydb/default/14 opened >>>>> in 68.922787ms >>>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL starting with 10485760 segment >>>>> size >>>>> [tsm1wal] 2016/08/01 12:45:40 tsm1 WAL writing to >>>>> /data/influxdb/wal/mydb/default/15 >>>>> panic: runtime error: slice bounds out of range >>>>> >>>>> goroutine 496 [running]: >>>>> panic(0xb5b240, 0xc82000e040) >>>>> /usr/local/go/src/runtime/panic.go:481 +0x3e6 >>>>> >>>>> github.com/influxdata/influxdb/tsdb/engine/tsm1.(*mmapAccessor).init(0xc8235f78c0, >>>>> 0x0, 0x0, 0x0) >>>>> /root/go/src/ >>>>> github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:879 +0x5a8 >>>>> >>>>> github.com/influxdata/influxdb/tsdb/engine/tsm1.NewTSMReader(0xc825842030, >>>>> 0xc80b8156de, 0x0, 0x0) >>>>> /root/go/src/ >>>>> github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:182 +0x204 >>>>> >>>>> github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Open.func1(0xc824c1c180, >>>>> 0xc8236cb080, 0x2, 0xc825842030) >>>>> /root/go/src/ >>>>> github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:332 >>>>> +0x77 >>>>> created by >>>>> github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Open >>>>> /root/go/src/ >>>>> github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:342 >>>>> +0x749 >>>>> >>>> -- >> Remember to include the InfluxDB version number with all issue reports >> --- >> You received this message because you are subscribed to the Google Groups >> "InfluxDB" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/influxdb. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/influxdb/5f0edf0a-227b-436d-bc82-05332f53c191%40googlegroups.com >> <https://groups.google.com/d/msgid/influxdb/5f0edf0a-227b-436d-bc82-05332f53c191%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Sean Beckett > Director of Support and Professional Services > InfluxDB > -- Sean Beckett Director of Support and Professional Services InfluxDB -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CALGqCvNw2DXk006jb1LT9nZB9pHBohHqbROQkDyGSopK0md5Lw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
