When no explicit timestamp is supplied, InfluxDB applies the same timestamp to all points in the batch. Since all points belong to the same series, InfluxDB treats them as duplicate points, and stores the union of the points, with the last field winning.
https://docs.influxdata.com/influxdb/v0.13/troubleshooting/frequently_encountered_issues/#writing-duplicate-points To avoid this, you need to explicitly supply timestamps (preferred) or write the points in separate batches so they will inherit unique nanosecond timestamps. Note that not batching writes will lead to significantly reduced write throughput, due to the high HTTP overhead. On Wed, Sep 7, 2016 at 7:46 AM, indranilr <[email protected]> wrote: > I am consuming messages from per-populated Kafka topic and trying to POST > to influxdb (tried via telegraf and node.js client) but seeing that > multiple POSTs issued in short span of time is not getting reflected in > influxdb database.I wasn't sure if this is an actual bug or an issue with > some configuration in my code.Kindly advise. > > Following are the points I have populated in Kafka queue : > > {"value":40,"host":"host1"} > {"value":50,"host":"host1"} > {"value":60,"host":"host1"} > {"value":70,"host":"host1"} > {"value":80,"host":"host1"} > Once node.js client consumes (using kafka-node) the same it writes it to > influx (using node-influx) using *writePoint *method (post some parsing) > > influxClient.writePoint(target_metric, {value : vl}, {host : ht} , {db: > db_name},callback); > > Please note* no timestamp* is being sent to influx. > > However, when I check in target influx measurements I only fewer points > than originally sent (you can see only 4 points below). > > name: test_ms_1 > --------------- > time host value > 1473251241835000000 host1 40 > 1473251241838000000 host1 80 > 1473251241860000000 host1 60 > 1473251241861000000 host1 50 > > > Could anyone explain the above behavior is due to missing timestamp in > input data or something else is wrong. > > I checked influx log and I can see equivalent number of POST queries and > relevant message bodies (I have turned on body tracing) > > [httpd] 2016/09/07 17:57:21 write body received by handler: >> test_ms_1,host=host1 value=40 >> [tsm1wal] 2016/09/07 17:57:21 tsm1 WAL starting with 10485760 segment size >> [tsm1wal] 2016/09/07 17:57:21 tsm1 WAL writing to >> /var/lib/influxdb/wal/eridb6/default/12 >> [shard] 2016/09/07 17:57:21 /var/lib/influxdb/data/eridb6/default/12 >> database index loaded in 986ns >> [httpd] 2016/09/07 17:57:21 write body received by handler: >> test_ms_1,host=host1 value=80 >> [httpd] 2016/09/07 17:57:21 write body received by handler: >> test_ms_1,host=host1 value=70 >> [httpd] 2016/09/07 17:57:21 write body received by handler: >> test_ms_1,host=host1 value=60 >> [httpd] 2016/09/07 17:57:21 write body received by handler: >> test_ms_1,host=host1 value=50 >> [httpd] 2016/09/07 17:57:21 10.0.2.2 - root [07/Sep/2016:17:57:21 +0530] >> POST /write?database=&db=eridb6&p=%5BREDACTED%5D&precision=ms&u=root >> HTTP/1.1 204 0 - - 6cd985c1-74f6-11e6-8013-000000000000 26.071054ms >> [httpd] 2016/09/07 17:57:21 10.0.2.2 - root [07/Sep/2016:17:57:21 +0530] >> POST /write?database=&db=eridb6&p=%5BREDACTED%5D&precision=ms&u=root >> HTTP/1.1 204 0 - - 6cda049b-74f6-11e6-8014-000000000000 29.699655ms >> [httpd] 2016/09/07 17:57:21 10.0.2.2 - root [07/Sep/2016:17:57:21 +0530] >> POST /write?database=&db=eridb6&p=%5BREDACTED%5D&precision=ms&u=root >> HTTP/1.1 204 0 - - 6cdd57e2-74f6-11e6-8015-000000000000 10.920647ms >> [httpd] 2016/09/07 17:57:21 10.0.2.2 - root [07/Sep/2016:17:57:21 +0530] >> POST /write?database=&db=eridb6&p=%5BREDACTED%5D&precision=ms&u=root >> HTTP/1.1 204 0 - - 6cdd765b-74f6-11e6-8016-000000000000 11.097619ms > > [httpd] 2016/09/07 17:57:21 10.0.2.2 - root [07/Sep/2016:17:57:21 +0530] >> POST /write?database=&db=eridb6&p=%5BREDACTED%5D&precision=ms&u=root >> HTTP/1.1 204 0 - - 6cdd7c1a-74f6-11e6-8017-000000000000 11.845628ms >> >> > I am using influx 0.13 on CentOS Linux release 7.2.1511 (Core) (64 bit) on > Virtual box: > > Static hostname: localhost.centos > Icon name: computer-vm > Chassis: vm > Machine ID: XXXXXXXXXX > Boot ID: XXXXXXXXXXXXXXXX > Virtualization: kvm > Operating System: CentOS Linux 7 (Core) > CPE OS Name: cpe:/o:centos:centos:7 > Kernel: Linux 3.10.0-327.el7.x86_64 > Architecture: x86-64 > > System resources : > > *Memory :* > > MemTotal: 1016860 kB > MemFree: 699836 kB > MemAvailable: 716180 kB > Cached: 126384 kB > SwapCached: 0 kB > SwapTotal: 839676 kB > SwapFree: 839676 kB > > > *CPU:* > > cache size : 3072 KB > physical id : 0 > siblings : 1 > core id : 0 > cpu cores : 1 > apicid : 0 > initial apicid : 0 > fpu : yes > fpu_exception : yes > cpuid level : 13 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge > mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm > constant_tsc rep_good nopl xtopology nonstop_tsc pni pclmulqdq monitor > ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx rdrand hypervisor > lahf_lm abm > bogomips : 5188.00 > clflush size : 64 > cache_alignment : 64 > address sizes : 39 bits physical, 48 bits virtual > > > -- > Remember to include the InfluxDB version number with all issue reports > --- > You received this message because you are subscribed to the Google Groups > "InfluxDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/influxdb. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/influxdb/e42d40c9-f743-44ab-b2d9-7a0c18a6bfeb%40googlegroups.com > <https://groups.google.com/d/msgid/influxdb/e42d40c9-f743-44ab-b2d9-7a0c18a6bfeb%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Sean Beckett Director of Support and Professional Services InfluxDB -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CALGqCvNTvr9QMgFrYtj2yhW1NNKztEdehpCMXWcxJ4xineGMdg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
