Re: [influxdb] Internal server error, timeout and unusable server after large imports

[Tanya Unterberger]

Thanks, Sean.

It is good to know what the limitations are. And good that I made a mistake
at the start and we kind of have a work around...

On 13 October 2016 at 16:24, Sean Beckett <s...@influxdb.com> wrote:

> Tonya, when you write the data in ms but don't specify the precision, the
> database interprets those millisecond timestamps as nanoseconds, and all
> the data is written to a single shard covering Jan 1, 1970.
>
>
> > insert msns value=42 147633619
> > select * from msns
> name: msns
> --
> time value
> 147633619 42
>
> > precision rfc3339
> > select * from msns
> name: msns
> --
> time value
> 1970-01-01T00:24:36.33619Z 42
>
> That's why everything is fast, because all the data is in one shard.
>
> On Wed, Oct 12, 2016 at 9:50 PM, Tanya Unterberger <
> tanya.unterber...@gmail.com> wrote:
>
>> Hi Sean,
>>
>> I can reproduce all the CPU issues, slowness, etc. if I try to import the
>> data that I have in milliseconds, specifying precision as milliseconds.
>>
>> If I insert the same data without specifying any precision and query
>> without specifying any precision, the database is lightingly fast. The same
>> data.
>>
>> The reason I was adding precision=ms is that I thought it was the right
>> thing to do. The manual advises that Influx sores the data in nanoseconds
>> but to use the lowest precision to insert. So at some stage I even used
>> hours, but inserted the data with precision=h. When Influx tried to convert
>> that data to nanoseconds, index, etc, then it was having a hissy fit.
>>
>> Is it a bug or the manual should state that if you query the data at the
>> same precision as you insert, then you can go with the lowest precision and
>> do not specify what precision you are inserting?
>>
>> Thanks,
>> Tanya
>>
>> On 13 October 2016 at 10:26, Tanya Unterberger <
>> tanya.unterber...@gmail.com> wrote:
>>
>>> Hi Sean,
>>>
>>> The data is from 1838 to 2016, daily (sparse at times). We need to
>>> retain it, therefore the default policy.
>>>
>>> Thanks,
>>> Tanya
>>>
>>> On 13 October 2016 at 06:26, Sean Beckett <s...@influxdb.com> wrote:
>>>
>>>> Tanya, what range of time does your data cover? What are the retention
>>>> policies on the database?
>>>>
>>>> On Tue, Oct 11, 2016 at 11:14 PM, Tanya Unterberger <
>>>> tanya.unterber...@gmail.com> wrote:
>>>>
>>>>> Hi Sean,
>>>>>
>>>>> 1. Initially I killed the process
>>>>> 2. At some point I restarted influxdb service
>>>>> 3. Error logs show no errors
>>>>> 4. I rebuilt the server, installed the latest rpm. Reimported the data
>>>>> via scripts. Data goes in, but the server is unusable. Looks like indexing
>>>>> might be stuffed. The size of the data in that database is 38M. Total size
>>>>> of /var/lib/influxdb/data/ 273M
>>>>> 5. CPU went beserk and doesn't come down
>>>>> 6. A query like select count(blah) to the measurement that was batch
>>>>> inserted (10k records at a time) is unusable and times out
>>>>> 7. I need to import around 15 million records. How should I throttle
>>>>> that?
>>>>>
>>>>> At the moment I am pulling my hair out (not a pretty sight)
>>>>>
>>>>> Thanks a lot!
>>>>> Tanya
>>>>>
>>>>> On 12 October 2016 at 06:11, Sean Beckett <s...@influxdb.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 11, 2016 at 12:11 AM, <tanya.unterber...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> It seems that the old issue might have surfaced again (#3349) in
>>>>>>> v1.0.
>>>>>>>
>>>>>>> I tried to insert a large number of records (3913595) via a script,
>>>>>>> inserting 1 rows at a time.
>>>>>>>
>>>>>>> After a while I received
>>>>>>>
>>>>>>> HTTP/1.1 500 Internal Server Error
>>>>>>> Content-Type: application/json
>>>>>>> Request-Id: ac8ebbbe-8f70-11e6-8ce7-
>>>>>>> X-Influxdb-Version: 1.0.0
>>>>>>> Date: Tue, 11 Oc

Re: [influxdb] Internal server error, timeout and unusable server after large imports

[Tanya Unterberger]

Hi Sean,

I can reproduce all the CPU issues, slowness, etc. if I try to import the
data that I have in milliseconds, specifying precision as milliseconds.

If I insert the same data without specifying any precision and query
without specifying any precision, the database is lightingly fast. The same
data.

The reason I was adding precision=ms is that I thought it was the right
thing to do. The manual advises that Influx sores the data in nanoseconds
but to use the lowest precision to insert. So at some stage I even used
hours, but inserted the data with precision=h. When Influx tried to convert
that data to nanoseconds, index, etc, then it was having a hissy fit.

Is it a bug or the manual should state that if you query the data at the
same precision as you insert, then you can go with the lowest precision and
do not specify what precision you are inserting?

Thanks,
Tanya

On 13 October 2016 at 10:26, Tanya Unterberger <tanya.unterber...@gmail.com>
wrote:

> Hi Sean,
>
> The data is from 1838 to 2016, daily (sparse at times). We need to retain
> it, therefore the default policy.
>
> Thanks,
> Tanya
>
> On 13 October 2016 at 06:26, Sean Beckett <s...@influxdb.com> wrote:
>
>> Tanya, what range of time does your data cover? What are the retention
>> policies on the database?
>>
>> On Tue, Oct 11, 2016 at 11:14 PM, Tanya Unterberger <
>> tanya.unterber...@gmail.com> wrote:
>>
>>> Hi Sean,
>>>
>>> 1. Initially I killed the process
>>> 2. At some point I restarted influxdb service
>>> 3. Error logs show no errors
>>> 4. I rebuilt the server, installed the latest rpm. Reimported the data
>>> via scripts. Data goes in, but the server is unusable. Looks like indexing
>>> might be stuffed. The size of the data in that database is 38M. Total size
>>> of /var/lib/influxdb/data/ 273M
>>> 5. CPU went beserk and doesn't come down
>>> 6. A query like select count(blah) to the measurement that was batch
>>> inserted (10k records at a time) is unusable and times out
>>> 7. I need to import around 15 million records. How should I throttle
>>> that?
>>>
>>> At the moment I am pulling my hair out (not a pretty sight)
>>>
>>> Thanks a lot!
>>> Tanya
>>>
>>> On 12 October 2016 at 06:11, Sean Beckett <s...@influxdb.com> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Oct 11, 2016 at 12:11 AM, <tanya.unterber...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> It seems that the old issue might have surfaced again (#3349) in v1.0.
>>>>>
>>>>> I tried to insert a large number of records (3913595) via a script,
>>>>> inserting 1 rows at a time.
>>>>>
>>>>> After a while I received
>>>>>
>>>>> HTTP/1.1 500 Internal Server Error
>>>>> Content-Type: application/json
>>>>> Request-Id: ac8ebbbe-8f70-11e6-8ce7-
>>>>> X-Influxdb-Version: 1.0.0
>>>>> Date: Tue, 11 Oct 2016 05:12:02 GMT
>>>>> Content-Length: 20
>>>>>
>>>>> {"error":"timeout"}
>>>>> HTTP/1.1 100 Continue
>>>>>
>>>>> I killed the process, after which the whole box became pretty much
>>>>> unresponsive.
>>>>>
>>>>
>>>> Killed the InfluxDB process, or the batch writing script process?
>>>>
>>>>
>>>>>
>>>>> There is nothing in the logs (i.e. sudo ls /var/log/influxdb/ gives me
>>>>> nothing) although the setting for http logging is true:
>>>>>
>>>>
>>>> systemd OSes put the logs in a new place (yay!?). See
>>>> http://docs.influxdata.com/influxdb/v1.0/administration/logs/#systemd
>>>> for how to read the logs.
>>>>
>>>>
>>>>>
>>>>> [http]
>>>>>   enabled = true
>>>>>   bind-address = ":8086"
>>>>>   auth-enabled = true
>>>>>   log-enabled = true
>>>>>
>>>>> I tried to restart influx, but got the following error:
>>>>>
>>>>> Failed to connect to http://localhost:8086
>>>>> Please check your connection settings and ensure 'influxd' is running.
>>>>>
>>>>
>>>> The `influx` console is just a fancy wrapper on the API. That error
>>>> doesn't mean much except that the HTTP listener in InfluxDB is not yet u

Re: [influxdb] Internal server error, timeout and unusable server after large imports

[Tanya Unterberger]

Hi Sean,

The data is from 1838 to 2016, daily (sparse at times). We need to retain
it, therefore the default policy.

Thanks,
Tanya

On 13 October 2016 at 06:26, Sean Beckett <s...@influxdb.com> wrote:

> Tanya, what range of time does your data cover? What are the retention
> policies on the database?
>
> On Tue, Oct 11, 2016 at 11:14 PM, Tanya Unterberger <
> tanya.unterber...@gmail.com> wrote:
>
>> Hi Sean,
>>
>> 1. Initially I killed the process
>> 2. At some point I restarted influxdb service
>> 3. Error logs show no errors
>> 4. I rebuilt the server, installed the latest rpm. Reimported the data
>> via scripts. Data goes in, but the server is unusable. Looks like indexing
>> might be stuffed. The size of the data in that database is 38M. Total size
>> of /var/lib/influxdb/data/ 273M
>> 5. CPU went beserk and doesn't come down
>> 6. A query like select count(blah) to the measurement that was batch
>> inserted (10k records at a time) is unusable and times out
>> 7. I need to import around 15 million records. How should I throttle that?
>>
>> At the moment I am pulling my hair out (not a pretty sight)
>>
>> Thanks a lot!
>> Tanya
>>
>> On 12 October 2016 at 06:11, Sean Beckett <s...@influxdb.com> wrote:
>>
>>>
>>>
>>> On Tue, Oct 11, 2016 at 12:11 AM, <tanya.unterber...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> It seems that the old issue might have surfaced again (#3349) in v1.0.
>>>>
>>>> I tried to insert a large number of records (3913595) via a script,
>>>> inserting 1 rows at a time.
>>>>
>>>> After a while I received
>>>>
>>>> HTTP/1.1 500 Internal Server Error
>>>> Content-Type: application/json
>>>> Request-Id: ac8ebbbe-8f70-11e6-8ce7-
>>>> X-Influxdb-Version: 1.0.0
>>>> Date: Tue, 11 Oct 2016 05:12:02 GMT
>>>> Content-Length: 20
>>>>
>>>> {"error":"timeout"}
>>>> HTTP/1.1 100 Continue
>>>>
>>>> I killed the process, after which the whole box became pretty much
>>>> unresponsive.
>>>>
>>>
>>> Killed the InfluxDB process, or the batch writing script process?
>>>
>>>
>>>>
>>>> There is nothing in the logs (i.e. sudo ls /var/log/influxdb/ gives me
>>>> nothing) although the setting for http logging is true:
>>>>
>>>
>>> systemd OSes put the logs in a new place (yay!?). See
>>> http://docs.influxdata.com/influxdb/v1.0/administration/logs/#systemd
>>> for how to read the logs.
>>>
>>>
>>>>
>>>> [http]
>>>>   enabled = true
>>>>   bind-address = ":8086"
>>>>   auth-enabled = true
>>>>   log-enabled = true
>>>>
>>>> I tried to restart influx, but got the following error:
>>>>
>>>> Failed to connect to http://localhost:8086
>>>> Please check your connection settings and ensure 'influxd' is running.
>>>>
>>>
>>> The `influx` console is just a fancy wrapper on the API. That error
>>> doesn't mean much except that the HTTP listener in InfluxDB is not yet up
>>> and running.
>>>
>>>
>>>>
>>>> Although I can see that influxd is up an running:
>>>>
>>>> > systemctl | grep influx
>>>> influxdb.service
>>>>   loaded active running   InfluxDB is an open-source,
>>>> distributed, time series database
>>>>
>>>> What do I do now?
>>>>
>>>
>>> Check the logs as referenced above.
>>>
>>> The non-responsiveness on startup isn't surprising. It sounds like the
>>> system was overwhelmed with writes, which means that the WAL would have
>>> many points cached, waiting to be flushed to disk. On restart, InfluxDB
>>> won't accept new writes or queries until the cached ones in the WAL have
>>> persisted. For this reason, the HTTP listener is off until the WAL is
>>> flushed.
>>>
>>>
>>>>
>>>> I tried the same import over the weekend, then the script timeout
>>>> happened eventually but the result was the same unresponsive, unusable
>>>> server. We rebuilt the box and started again.
>>>>
>>>
>>> It sounds like the box is just overwhelmed. Did you get backoff messages
>>&

Re: [influxdb] Internal server error, timeout and unusable server after large imports

[Tanya Unterberger]

Hi Sean,

1. Initially I killed the process
2. At some point I restarted influxdb service
3. Error logs show no errors
4. I rebuilt the server, installed the latest rpm. Reimported the data via
scripts. Data goes in, but the server is unusable. Looks like indexing
might be stuffed. The size of the data in that database is 38M. Total size
of /var/lib/influxdb/data/ 273M
5. CPU went beserk and doesn't come down
6. A query like select count(blah) to the measurement that was batch
inserted (10k records at a time) is unusable and times out
7. I need to import around 15 million records. How should I throttle that?

At the moment I am pulling my hair out (not a pretty sight)

Thanks a lot!
Tanya

On 12 October 2016 at 06:11, Sean Beckett  wrote:

>
>
> On Tue, Oct 11, 2016 at 12:11 AM,  wrote:
>
>> Hi,
>>
>> It seems that the old issue might have surfaced again (#3349) in v1.0.
>>
>> I tried to insert a large number of records (3913595) via a script,
>> inserting 1 rows at a time.
>>
>> After a while I received
>>
>> HTTP/1.1 500 Internal Server Error
>> Content-Type: application/json
>> Request-Id: ac8ebbbe-8f70-11e6-8ce7-
>> X-Influxdb-Version: 1.0.0
>> Date: Tue, 11 Oct 2016 05:12:02 GMT
>> Content-Length: 20
>>
>> {"error":"timeout"}
>> HTTP/1.1 100 Continue
>>
>> I killed the process, after which the whole box became pretty much
>> unresponsive.
>>
>
> Killed the InfluxDB process, or the batch writing script process?
>
>
>>
>> There is nothing in the logs (i.e. sudo ls /var/log/influxdb/ gives me
>> nothing) although the setting for http logging is true:
>>
>
> systemd OSes put the logs in a new place (yay!?). See
> http://docs.influxdata.com/influxdb/v1.0/administration/logs/#systemd for
> how to read the logs.
>
>
>>
>> [http]
>>   enabled = true
>>   bind-address = ":8086"
>>   auth-enabled = true
>>   log-enabled = true
>>
>> I tried to restart influx, but got the following error:
>>
>> Failed to connect to http://localhost:8086
>> Please check your connection settings and ensure 'influxd' is running.
>>
>
> The `influx` console is just a fancy wrapper on the API. That error
> doesn't mean much except that the HTTP listener in InfluxDB is not yet up
> and running.
>
>
>>
>> Although I can see that influxd is up an running:
>>
>> > systemctl | grep influx
>> influxdb.service
>> loaded active running   InfluxDB is an open-source,
>> distributed, time series database
>>
>> What do I do now?
>>
>
> Check the logs as referenced above.
>
> The non-responsiveness on startup isn't surprising. It sounds like the
> system was overwhelmed with writes, which means that the WAL would have
> many points cached, waiting to be flushed to disk. On restart, InfluxDB
> won't accept new writes or queries until the cached ones in the WAL have
> persisted. For this reason, the HTTP listener is off until the WAL is
> flushed.
>
>
>>
>> I tried the same import over the weekend, then the script timeout
>> happened eventually but the result was the same unresponsive, unusable
>> server. We rebuilt the box and started again.
>>
>
> It sounds like the box is just overwhelmed. Did you get backoff messages
> from the writes before the crash? What are the machine specs?
>
>
>
>>
>> Perhaps it is worthwhile mentioning that the same measurement already
>> contained about 9 million records. Some of these records had the same
>> timestamp as the ones I tried to import, i.e. they should have been merged.
>>
>
> Overwriting points is much much more expensive than posting new points.
> Each overwritten point triggers a tombstone record which must later be
> processed. This can trigger frequent compactions of the TSM files. With a
> high write load and frequent compactions, the system would encounter
> significant CPU pressure.
>
>
>>
>> Interestingly enough the same amount of data was fine when I forgot to
>> add precision in ms, i.e. all records were imported as nanoseconds, but in
>> fact they "lacked" 6 zeroes.
>>
>
> That would mean all points are going to the same shard. It is more
> resource intensive to load points across a wide range of time, since more
> shard files are involved. InfluxDB does best with sequential
> chronologically ordered unique points from the very recent past. The more
> the write operation differs from that, the lower the throughput.
>
>
>>
>> Please advise what kind of action I can take.
>>
>
> Look in the logs for errors. Throttle the writes. Don't overwrite more
> points than you have to.
>
>
>>
>> Thanks a lot!
>> Tanya
>>
>> --
>> Remember to include the InfluxDB version number with all issue reports
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "InfluxDB" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to influxdb+unsubscr...@googlegroups.com.
>> To post to this group, send email to influxdb@googlegroups.com.
>> Visit this group at 

[influxdb] Internal server error, timeout and unusable server after large imports

[tanya . unterberger]

Hi, 

It seems that the old issue might have surfaced again (#3349) in v1.0.

I tried to insert a large number of records (3913595) via a script, inserting 
1 rows at a time. 

After a while I received 

HTTP/1.1 500 Internal Server Error
Content-Type: application/json
Request-Id: ac8ebbbe-8f70-11e6-8ce7-
X-Influxdb-Version: 1.0.0
Date: Tue, 11 Oct 2016 05:12:02 GMT
Content-Length: 20

{"error":"timeout"}
HTTP/1.1 100 Continue

I killed the process, after which the whole box became pretty much unresponsive.

There is nothing in the logs (i.e. sudo ls /var/log/influxdb/ gives me nothing) 
although the setting for http logging is true:

[http]
  enabled = true
  bind-address = ":8086"
  auth-enabled = true
  log-enabled = true

I tried to restart influx, but got the following error:

Failed to connect to http://localhost:8086
Please check your connection settings and ensure 'influxd' is running.

Although I can see that influxd is up an running:

> systemctl | grep influx
influxdb.service
  loaded active running   InfluxDB is an open-source, distributed, time 
series database

What do I do now? 

I tried the same import over the weekend, then the script timeout happened 
eventually but the result was the same unresponsive, unusable server. We 
rebuilt the box and started again.

Perhaps it is worthwhile mentioning that the same measurement already contained 
about 9 million records. Some of these records had the same timestamp as the 
ones I tried to import, i.e. they should have been merged.

Interestingly enough the same amount of data was fine when I forgot to add 
precision in ms, i.e. all records were imported as nanoseconds, but in fact 
they "lacked" 6 zeroes.

Please advise what kind of action I can take.

Thanks a lot!
Tanya

-- 
Remember to include the InfluxDB version number with all issue reports
--- 
You received this message because you are subscribed to the Google Groups 
"InfluxDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to influxdb+unsubscr...@googlegroups.com.
To post to this group, send email to influxdb@googlegroups.com.
Visit this group at https://groups.google.com/group/influxdb.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/influxdb/f4ebdb56-32f9-4fb6-88de-f7ef603c4262%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [influxdb] Descending ordering of series

[Tanya Unterberger]

Nope, sorry. I need the whole series and the latest series. Whereas the
query

select * from blah where x='y' and time > now() - 6h group by base_date
order by time desc limit 1 slimit 1

returns the last point of the earliest series.

On 5 October 2016 at 14:20, Sean Beckett <s...@influxdb.com> wrote:

> Does
>
> select * from blah where x='y' and time > now() - 6h group by base_date
> order by time desc limit 1 slimit 1
>
> return what you need?
>
> On Tue, Oct 4, 2016 at 8:59 PM, Tanya Unterberger <
> tanya.unterber...@gmail.com> wrote:
>
>> Thanks, Sean.
>>
>> But it's not the data I am after. I am after the latest series, not the
>> latest point in each series. That's why I was using SLIMIT, but it returns
>> the earliest series, whereas I need the latest...
>>
>> Tanya
>>
>> On 5 October 2016 at 12:32, Sean Beckett <s...@influxdb.com> wrote:
>>
>>> ORDER BY time DESC only orders each series, and then SLIMIT returns only
>>> one series. It doesn't necessarily return the series with the newest point.
>>> Just remove the SLIMIT and use LIMIT 1.
>>>
>>> Note that the LIMIT clause doesn't (yet) restrict the points queried.
>>> With a "SELECT *" clause it's important to restrict the time range.
>>>
>>> select * from blah where x='y' and time > now() - 6h group by base_date
>>> order by time desc limit 1
>>>
>>> On Tue, Oct 4, 2016 at 7:13 PM, Tanya Unterberger <
>>> tanya.unterber...@gmail.com> wrote:
>>>
>>>> Hi Sean,
>>>>
>>>> That doesn't work as it still grabs the earliest base_date, not the
>>>> latest...
>>>>
>>>> Thanks,
>>>> Tanya
>>>>
>>>> On 4 October 2016 at 03:24, Sean Beckett <s...@influxdb.com> wrote:
>>>>
>>>>> Add ORDER BY time DESC
>>>>> <https://docs.influxdata.com/influxdb/v1.0/query_language/data_exploration/#sort-query-returns-with-order-by-time-desc>
>>>>> to the query
>>>>>
>>>>> On Sun, Oct 2, 2016 at 6:49 PM, <tanya.unterber...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We have forecast data issued for the same period of time. It can be
>>>>>> issued at various time points, which we keep as base_date tags. What is 
>>>>>> the
>>>>>> easiest way to get the latest issued forecast? If you try to run 
>>>>>> something
>>>>>> like "select * from blah where x='y' group by base_date slimit 1" the
>>>>>> base_date is the earliest, not the latest.
>>>>>>
>>>>>> Thanks a lot,
>>>>>> Tanya
>>>>>>
>>>>>> --
>>>>>> Remember to include the InfluxDB version number with all issue reports
>>>>>> ---
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "InfluxDB" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to influxdb+unsubscr...@googlegroups.com.
>>>>>> To post to this group, send email to influxdb@googlegroups.com.
>>>>>> Visit this group at https://groups.google.com/group/influxdb.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/influxdb/f9f05733-bfdf-463
>>>>>> 7-900b-8e710ddfcd61%40googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sean Beckett
>>>>> Director of Support and Professional Services
>>>>> InfluxDB
>>>>>
>>>>> --
>>>>> Remember to include the InfluxDB version number with all issue reports
>>>>> ---
>>>>> You received this message because you are subscribed to a topic in the
>>>>> Google Groups "InfluxDB" group.
>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/to
>>>>> pic/influxdb/U-3Npsw6dZA/unsubscribe.
>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>> influxdb+unsubscr...@googlegroups.com.
>>>>> To post to this group, send email to influxdb@googlegroups.com.
>

Re: [influxdb] Can't insert records prior to 1970

[Tanya Unterberger]

Hi Sean,

Thank you for such a prompt reply. However, I still cannot query the data.

Again I ran:

> INSERT months,stn_num=66046
lsd="01/01/1832",prcp_day_cnt=1,prcp_day_cnt_qual="X",ttl_mo_prcp=55.9,ttl_mo_prcp_qual="X",data_src=1,last_update_prcp="23/07/2009"
-435490560

> SELECT * FROM month WHERE time > '1800-01-01'

> SELECT * FROM month WHERE time > now() - 500w


The time is 01/01/1832, so querying time >  '1800-01-01' should have worked?

Thanks,

Tanya

On 13 September 2016 at 02:18, Sean Beckett <s...@influxdb.com> wrote:

> Tanya,
>
> InfluxDB uses epoch 0 and now() as the default time boundaries for
> queries. You can imagine that every query has "WHERE time > 0 AND time <
> now()" in the where clause.
>
> When you use a query like "WHERE time < 0" that replaces the upper bound,
> but the implicit lower bound is still epoch 0. In effect, your query is
> this:
>
> SELECT * FROM month WHERE time > 0 AND time < 0
>
> As you can see, that will never have any results. To query before epoch 0
> you need to move the lower bound of the time range, not the upper bound.
> Try something like:
>
> SELECT * FROM month WHERE time > '1950-01-01'
> or
> SELECT * FROM month WHERE time > now() - 5000w
>
> On Mon, Sep 12, 2016 at 1:05 AM, Tanya Unterberger <
> tanya.unterber...@gmail.com> wrote:
>
>> Hi Sean,
>>
>> I have installed version 1.0.0. I still have an issue of querying data
>> prior to 1970.
>>
>> Am I doing anything wrong?
>>
>> My test is:
>>
>> > INSERT months,stn_num=66046 lsd="01/01/1832",prcp_day_cnt=
>> 1,prcp_day_cnt_qual="X",ttl_mo_prcp=55.9,ttl_mo_prcp_qual="
>> X",data_src=1,last_update_prcp="23/07/2009" -435490560
>>
>> > select * from month
>> > select * from month where time < 0
>>
>> Thanks,
>> Tanya
>>
>> On 30 August 2016 at 10:22, Tanya Unterberger <
>> tanya.unterber...@gmail.com> wrote:
>>
>>> Thank you!!!
>>>
>>> On 30 Aug 2016 9:28 AM, "Sean Beckett" <s...@influxdb.com> wrote:
>>>
>>>> Fixed and in the imminent RC2 release: https://github.com/in
>>>> fluxdata/influxdb/pull/7212
>>>>
>>>> On Tue, Aug 23, 2016 at 6:35 PM, Tanya Unterberger <
>>>> tanya.unterber...@gmail.com> wrote:
>>>>
>>>>> Great news, thanks a lot!!!
>>>>>
>>>>> On 24 August 2016 at 09:00, Sean Beckett <s...@influxdb.com> wrote:
>>>>>
>>>>>> It's a regression bug, so it is a priority. It's also fairly trivial,
>>>>>> so it should be quick. 1.0 will be released in early September and we 
>>>>>> just
>>>>>> cut RC1 today. I would anticipate this fix being in RC2, which will be 
>>>>>> out
>>>>>> later this week or early next.
>>>>>>
>>>>>> On Tue, Aug 23, 2016 at 4:51 PM, Tanya Unterberger <
>>>>>> tanya.unterber...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Sean,
>>>>>>>
>>>>>>> Thank you so much for the prompt reply, much appreciated!
>>>>>>>
>>>>>>> Is it a difficult bug to fix? I.e. when can we expect it to be
>>>>>>> fixed, please? I noticed that it was assigned to the version 1.0.0. Is
>>>>>>> there an ETA for this version, please? We have a large number of 
>>>>>>> historical
>>>>>>> records we need to insert and depending on whether we can query them or 
>>>>>>> not
>>>>>>> we would need to consider alternative solutions.
>>>>>>>
>>>>>>> Thanks again,
>>>>>>> Tanya
>>>>>>>
>>>>>>>
>>>>>>> On 24 August 2016 at 05:26, Sean Beckett <s...@influxdb.com> wrote:
>>>>>>>
>>>>>>>> Tanya, it appears to be a bug in 0.13 where the query engine won't
>>>>>>>> allow time ranges older than epoch 0. That is a regression bug and I've
>>>>>>>> opened an issue: https://github.com/influxdata/influxdb/issues/7194
>>>>>>>>
>>>>>>>> Your writes are succeeding and the data is persisting, but the data
>>>>>>>> is unqueryable until the bug is fixed.
>>>>>&

Re: [influxdb] Can't insert records prior to 1970

[Tanya Unterberger]

Hi Sean,

I have installed version 1.0.0. I still have an issue of querying data
prior to 1970.

Am I doing anything wrong?

My test is:

> INSERT months,stn_num=66046
lsd="01/01/1832",prcp_day_cnt=1,prcp_day_cnt_qual="X",ttl_mo_prcp=55.9,ttl_mo_prcp_qual="X",data_src=1,last_update_prcp="23/07/2009"
-435490560

> select * from month
> select * from month where time < 0

Thanks,
Tanya

On 30 August 2016 at 10:22, Tanya Unterberger <tanya.unterber...@gmail.com>
wrote:

> Thank you!!!
>
> On 30 Aug 2016 9:28 AM, "Sean Beckett" <s...@influxdb.com> wrote:
>
>> Fixed and in the imminent RC2 release: https://github.com/in
>> fluxdata/influxdb/pull/7212
>>
>> On Tue, Aug 23, 2016 at 6:35 PM, Tanya Unterberger <
>> tanya.unterber...@gmail.com> wrote:
>>
>>> Great news, thanks a lot!!!
>>>
>>> On 24 August 2016 at 09:00, Sean Beckett <s...@influxdb.com> wrote:
>>>
>>>> It's a regression bug, so it is a priority. It's also fairly trivial,
>>>> so it should be quick. 1.0 will be released in early September and we just
>>>> cut RC1 today. I would anticipate this fix being in RC2, which will be out
>>>> later this week or early next.
>>>>
>>>> On Tue, Aug 23, 2016 at 4:51 PM, Tanya Unterberger <
>>>> tanya.unterber...@gmail.com> wrote:
>>>>
>>>>> Hi Sean,
>>>>>
>>>>> Thank you so much for the prompt reply, much appreciated!
>>>>>
>>>>> Is it a difficult bug to fix? I.e. when can we expect it to be fixed,
>>>>> please? I noticed that it was assigned to the version 1.0.0. Is there an
>>>>> ETA for this version, please? We have a large number of historical records
>>>>> we need to insert and depending on whether we can query them or not we
>>>>> would need to consider alternative solutions.
>>>>>
>>>>> Thanks again,
>>>>> Tanya
>>>>>
>>>>>
>>>>> On 24 August 2016 at 05:26, Sean Beckett <s...@influxdb.com> wrote:
>>>>>
>>>>>> Tanya, it appears to be a bug in 0.13 where the query engine won't
>>>>>> allow time ranges older than epoch 0. That is a regression bug and I've
>>>>>> opened an issue: https://github.com/influxdata/influxdb/issues/7194
>>>>>>
>>>>>> Your writes are succeeding and the data is persisting, but the data
>>>>>> is unqueryable until the bug is fixed.
>>>>>>
>>>>>> The other queries are failing to return the results because the
>>>>>> default lower time bound is epoch 0. When you issue "SELECT * FROM
>>>>>> hist_days" there's an implicit "WHERE time > 0 and time < now()" on that
>>>>>> query due to internal defaults. The problem is that the query parser 
>>>>>> won't
>>>>>> currently allow a statement like "SELECT * FROM hist_days WHERE time >
>>>>>> '1900-01-01'", which would reset the lower bound.
>>>>>>
>>>>>> On Mon, Aug 22, 2016 at 10:16 PM, <tanya.unterber...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have a number or records that go back way before 1970. I am having
>>>>>>> trouble inserting these records into InfluxDB.
>>>>>>>
>>>>>>> I tried to insert very simple records either via CLI or via curl.
>>>>>>> Either approach does not give an error but the data doesn't get in.
>>>>>>> Alternatively I am not queuing the database correctly. In either case I
>>>>>>> cannot see the data I need to retrieve.
>>>>>>>
>>>>>>> The simplified version of the inserts and queries is as follows:
>>>>>>>
>>>>>>>
>>>>>>> [xxx ~]$ curl -i -XPOST 'http://localhost:8086/write?db=history' -H
>>>>>>> 'Content-Type: text/plain' --data-binary 'hist_days,stn_num=61310
>>>>>>> prcp=1,prcp_type="RAIN" -53136000'
>>>>>>> HTTP/1.1 204 No Content
>>>>>>> Request-Id: a5264040-68c4-11e6-80a3-
>>>>>>> X-Influxdb-Version: 0.13.0
>>>>>>> Date: Tue, 23 Aug 2016 00:00:47 GMT
>>>>>>>

[influxdb] Is there an equivalent of IN () for the SELECT queries

[tanya . unterberger]

Hi, 

I was wondering whether there is an equivalent of 

SELECT * FROM measurement WHERE tag in ('a', 'b', 'c')

I can write an OR query of course, but if there are more than a couple of 
options, it becomes quite tedious to dynamically construct a query.

Thanks :)

-- 
Remember to include the InfluxDB version number with all issue reports
--- 
You received this message because you are subscribed to the Google Groups 
"InfluxDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to influxdb+unsubscr...@googlegroups.com.
To post to this group, send email to influxdb@googlegroups.com.
Visit this group at https://groups.google.com/group/influxdb.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/influxdb/e2557f5a-ec7c-4f5b-9e17-782e8c774600%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [influxdb] Can't insert records prior to 1970

[Tanya Unterberger]

Thank you!!!

On 30 Aug 2016 9:28 AM, "Sean Beckett" <s...@influxdb.com> wrote:

> Fixed and in the imminent RC2 release: https://github.com/
> influxdata/influxdb/pull/7212
>
> On Tue, Aug 23, 2016 at 6:35 PM, Tanya Unterberger <
> tanya.unterber...@gmail.com> wrote:
>
>> Great news, thanks a lot!!!
>>
>> On 24 August 2016 at 09:00, Sean Beckett <s...@influxdb.com> wrote:
>>
>>> It's a regression bug, so it is a priority. It's also fairly trivial, so
>>> it should be quick. 1.0 will be released in early September and we just cut
>>> RC1 today. I would anticipate this fix being in RC2, which will be out
>>> later this week or early next.
>>>
>>> On Tue, Aug 23, 2016 at 4:51 PM, Tanya Unterberger <
>>> tanya.unterber...@gmail.com> wrote:
>>>
>>>> Hi Sean,
>>>>
>>>> Thank you so much for the prompt reply, much appreciated!
>>>>
>>>> Is it a difficult bug to fix? I.e. when can we expect it to be fixed,
>>>> please? I noticed that it was assigned to the version 1.0.0. Is there an
>>>> ETA for this version, please? We have a large number of historical records
>>>> we need to insert and depending on whether we can query them or not we
>>>> would need to consider alternative solutions.
>>>>
>>>> Thanks again,
>>>> Tanya
>>>>
>>>>
>>>> On 24 August 2016 at 05:26, Sean Beckett <s...@influxdb.com> wrote:
>>>>
>>>>> Tanya, it appears to be a bug in 0.13 where the query engine won't
>>>>> allow time ranges older than epoch 0. That is a regression bug and I've
>>>>> opened an issue: https://github.com/influxdata/influxdb/issues/7194
>>>>>
>>>>> Your writes are succeeding and the data is persisting, but the data is
>>>>> unqueryable until the bug is fixed.
>>>>>
>>>>> The other queries are failing to return the results because the
>>>>> default lower time bound is epoch 0. When you issue "SELECT * FROM
>>>>> hist_days" there's an implicit "WHERE time > 0 and time < now()" on that
>>>>> query due to internal defaults. The problem is that the query parser won't
>>>>> currently allow a statement like "SELECT * FROM hist_days WHERE time >
>>>>> '1900-01-01'", which would reset the lower bound.
>>>>>
>>>>> On Mon, Aug 22, 2016 at 10:16 PM, <tanya.unterber...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have a number or records that go back way before 1970. I am having
>>>>>> trouble inserting these records into InfluxDB.
>>>>>>
>>>>>> I tried to insert very simple records either via CLI or via curl.
>>>>>> Either approach does not give an error but the data doesn't get in.
>>>>>> Alternatively I am not queuing the database correctly. In either case I
>>>>>> cannot see the data I need to retrieve.
>>>>>>
>>>>>> The simplified version of the inserts and queries is as follows:
>>>>>>
>>>>>>
>>>>>> [xxx ~]$ curl -i -XPOST 'http://localhost:8086/write?db=history' -H
>>>>>> 'Content-Type: text/plain' --data-binary 'hist_days,stn_num=61310
>>>>>> prcp=1,prcp_type="RAIN" -53136000'
>>>>>> HTTP/1.1 204 No Content
>>>>>> Request-Id: a5264040-68c4-11e6-80a3-
>>>>>> X-Influxdb-Version: 0.13.0
>>>>>> Date: Tue, 23 Aug 2016 00:00:47 GMT
>>>>>>
>>>>>> [xxx ~]$ curl -i -XPOST 'http://localhost:8086/write?db=history' -H
>>>>>> 'Content-Type: text/plain' --data-binary 'hist_days,stn_num=61310
>>>>>> prcp=1,prcp_type="RAIN" 53136001'
>>>>>> HTTP/1.1 204 No Content
>>>>>> Request-Id: a7514208-68c4-11e6-80a4-
>>>>>> X-Influxdb-Version: 0.13.0
>>>>>> Date: Tue, 23 Aug 2016 00:00:51 GMT
>>>>>>
>>>>>> [xxx ~]$ influx
>>>>>> Visit https://enterprise.influxdata.com to register for updates,
>>>>>> InfluxDB server management, and monitoring.
>>>>>> Connected to http://localhost:8086 version 0.13.0
>>>>>> InfluxDB shell version: 0.13.0
>>>>>> >

Re: [influxdb] Can't insert records prior to 1970

[Tanya Unterberger]

Great news, thanks a lot!!!

On 24 August 2016 at 09:00, Sean Beckett <s...@influxdb.com> wrote:

> It's a regression bug, so it is a priority. It's also fairly trivial, so
> it should be quick. 1.0 will be released in early September and we just cut
> RC1 today. I would anticipate this fix being in RC2, which will be out
> later this week or early next.
>
> On Tue, Aug 23, 2016 at 4:51 PM, Tanya Unterberger <
> tanya.unterber...@gmail.com> wrote:
>
>> Hi Sean,
>>
>> Thank you so much for the prompt reply, much appreciated!
>>
>> Is it a difficult bug to fix? I.e. when can we expect it to be fixed,
>> please? I noticed that it was assigned to the version 1.0.0. Is there an
>> ETA for this version, please? We have a large number of historical records
>> we need to insert and depending on whether we can query them or not we
>> would need to consider alternative solutions.
>>
>> Thanks again,
>> Tanya
>>
>>
>> On 24 August 2016 at 05:26, Sean Beckett <s...@influxdb.com> wrote:
>>
>>> Tanya, it appears to be a bug in 0.13 where the query engine won't allow
>>> time ranges older than epoch 0. That is a regression bug and I've opened an
>>> issue: https://github.com/influxdata/influxdb/issues/7194
>>>
>>> Your writes are succeeding and the data is persisting, but the data is
>>> unqueryable until the bug is fixed.
>>>
>>> The other queries are failing to return the results because the default
>>> lower time bound is epoch 0. When you issue "SELECT * FROM hist_days"
>>> there's an implicit "WHERE time > 0 and time < now()" on that query due to
>>> internal defaults. The problem is that the query parser won't currently
>>> allow a statement like "SELECT * FROM hist_days WHERE time > '1900-01-01'",
>>> which would reset the lower bound.
>>>
>>> On Mon, Aug 22, 2016 at 10:16 PM, <tanya.unterber...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a number or records that go back way before 1970. I am having
>>>> trouble inserting these records into InfluxDB.
>>>>
>>>> I tried to insert very simple records either via CLI or via curl.
>>>> Either approach does not give an error but the data doesn't get in.
>>>> Alternatively I am not queuing the database correctly. In either case I
>>>> cannot see the data I need to retrieve.
>>>>
>>>> The simplified version of the inserts and queries is as follows:
>>>>
>>>>
>>>> [xxx ~]$ curl -i -XPOST 'http://localhost:8086/write?db=history' -H
>>>> 'Content-Type: text/plain' --data-binary 'hist_days,stn_num=61310
>>>> prcp=1,prcp_type="RAIN" -53136000'
>>>> HTTP/1.1 204 No Content
>>>> Request-Id: a5264040-68c4-11e6-80a3-
>>>> X-Influxdb-Version: 0.13.0
>>>> Date: Tue, 23 Aug 2016 00:00:47 GMT
>>>>
>>>> [xxx ~]$ curl -i -XPOST 'http://localhost:8086/write?db=history' -H
>>>> 'Content-Type: text/plain' --data-binary 'hist_days,stn_num=61310
>>>> prcp=1,prcp_type="RAIN" 53136001'
>>>> HTTP/1.1 204 No Content
>>>> Request-Id: a7514208-68c4-11e6-80a4-
>>>> X-Influxdb-Version: 0.13.0
>>>> Date: Tue, 23 Aug 2016 00:00:51 GMT
>>>>
>>>> [xxx ~]$ influx
>>>> Visit https://enterprise.influxdata.com to register for updates,
>>>> InfluxDB server management, and monitoring.
>>>> Connected to http://localhost:8086 version 0.13.0
>>>> InfluxDB shell version: 0.13.0
>>>> > use history
>>>> Using database history
>>>> > select * from hist_days
>>>> name: hist_days
>>>> ---
>>>> timeprcpprcp_type   stn_num
>>>> 531360011   RAIN61310
>>>>
>>>> > select * from hist_days where time < 0
>>>> >
>>>>
>>>>
>>>> Can you please help as I need to ingest historical data!!!
>>>>
>>>> Tanks a lot, any help is appreciated.
>>>>
>>>> Tanya
>>>>
>>>> --
>>>> Remember to include the InfluxDB version number with all issue reports
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "InfluxDB" group.
>>>> To unsubscribe from this group and stop

Re: [influxdb] Can't insert records prior to 1970

[Tanya Unterberger]

Hi Sean,

Thank you so much for the prompt reply, much appreciated!

Is it a difficult bug to fix? I.e. when can we expect it to be fixed,
please? I noticed that it was assigned to the version 1.0.0. Is there an
ETA for this version, please? We have a large number of historical records
we need to insert and depending on whether we can query them or not we
would need to consider alternative solutions.

Thanks again,
Tanya


On 24 August 2016 at 05:26, Sean Beckett  wrote:

> Tanya, it appears to be a bug in 0.13 where the query engine won't allow
> time ranges older than epoch 0. That is a regression bug and I've opened an
> issue: https://github.com/influxdata/influxdb/issues/7194
>
> Your writes are succeeding and the data is persisting, but the data is
> unqueryable until the bug is fixed.
>
> The other queries are failing to return the results because the default
> lower time bound is epoch 0. When you issue "SELECT * FROM hist_days"
> there's an implicit "WHERE time > 0 and time < now()" on that query due to
> internal defaults. The problem is that the query parser won't currently
> allow a statement like "SELECT * FROM hist_days WHERE time > '1900-01-01'",
> which would reset the lower bound.
>
> On Mon, Aug 22, 2016 at 10:16 PM,  wrote:
>
>> Hi,
>>
>> I have a number or records that go back way before 1970. I am having
>> trouble inserting these records into InfluxDB.
>>
>> I tried to insert very simple records either via CLI or via curl. Either
>> approach does not give an error but the data doesn't get in. Alternatively
>> I am not queuing the database correctly. In either case I cannot see the
>> data I need to retrieve.
>>
>> The simplified version of the inserts and queries is as follows:
>>
>>
>> [xxx ~]$ curl -i -XPOST 'http://localhost:8086/write?db=history' -H
>> 'Content-Type: text/plain' --data-binary 'hist_days,stn_num=61310
>> prcp=1,prcp_type="RAIN" -53136000'
>> HTTP/1.1 204 No Content
>> Request-Id: a5264040-68c4-11e6-80a3-
>> X-Influxdb-Version: 0.13.0
>> Date: Tue, 23 Aug 2016 00:00:47 GMT
>>
>> [xxx ~]$ curl -i -XPOST 'http://localhost:8086/write?db=history' -H
>> 'Content-Type: text/plain' --data-binary 'hist_days,stn_num=61310
>> prcp=1,prcp_type="RAIN" 53136001'
>> HTTP/1.1 204 No Content
>> Request-Id: a7514208-68c4-11e6-80a4-
>> X-Influxdb-Version: 0.13.0
>> Date: Tue, 23 Aug 2016 00:00:51 GMT
>>
>> [xxx ~]$ influx
>> Visit https://enterprise.influxdata.com to register for updates,
>> InfluxDB server management, and monitoring.
>> Connected to http://localhost:8086 version 0.13.0
>> InfluxDB shell version: 0.13.0
>> > use history
>> Using database history
>> > select * from hist_days
>> name: hist_days
>> ---
>> timeprcpprcp_type   stn_num
>> 531360011   RAIN61310
>>
>> > select * from hist_days where time < 0
>> >
>>
>>
>> Can you please help as I need to ingest historical data!!!
>>
>> Tanks a lot, any help is appreciated.
>>
>> Tanya
>>
>> --
>> Remember to include the InfluxDB version number with all issue reports
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "InfluxDB" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to influxdb+unsubscr...@googlegroups.com.
>> To post to this group, send email to influxdb@googlegroups.com.
>> Visit this group at https://groups.google.com/group/influxdb.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/influxdb/61366ffc-2c9c-4b49-8599-28de3b2b7332%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Sean Beckett
> Director of Support and Professional Services
> InfluxDB
>
> --
> Remember to include the InfluxDB version number with all issue reports
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "InfluxDB" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/influxdb/JSSmSp5oBaU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> influxdb+unsubscr...@googlegroups.com.
> To post to this group, send email to influxdb@googlegroups.com.
> Visit this group at https://groups.google.com/group/influxdb.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/influxdb/CALGqCvPvUDGidNvzt032x0v%2BmJyfjRaUnshBDgDSZV9HVPqbnA%
> 40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Remember to include the InfluxDB version number with all issue reports
--- 
You received this message because you are subscribed to the Google Groups 
"InfluxDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to