[weewx-user] Re: Importing data from a CSV file

2017-04-22 Thread MRL 
I am about to import 400,000 records contained in 9 files. I did dry runs 
and no errors. This is 9 months of Weather-Display data in one-minute 
intervals.
I would like to make a copy of the database before I start. I have used 
Sqlite before. Not sure how to use your schema to do this. Can I just copy 
the database and rename it? Do I need to stop weewx before copying the 
database?  I might wait until the current weather stabilizes before 
stopping weewx.
Thanks.

On Friday, April 21, 2017 at 10:31:05 AM UTC-4, MRL wrote:
>
> I am ready to do a dry run.
> It took a trivial python program to convert:
> day month year hour minute temperature   humidity dewpoint   barometer 
>windspeed   gustspeed direction  rainlastmindailyrain  monthlyrain   
> yearlyrain  heatindex
>  1  1 2017  0  1 44.1  46 24.6 29.953 0.2 0.0 245  0.000 0.000 0.000 0.000 
> 44.1
>  1  1 2017  0  2 44.1  46 24.6 29.953 0.0 0.0  60  0.000 0.000 0.000 0.000 
> 44.1
>  1  1 2017  0  3 44.1  46 24.6 29.953 0.3 3.0  12  0.000 0.000 0.000 0.000 
> 44.1
> into:
>
> datetime,temperature,humidity,dewpoint,barometer,windspeed,gustspeed,direction,rainlastmin,dailyrain,monthlyrain,yearlyrain,heatindex
> 2017-01-01 
> 00:01:00,44.1,46,24.6,29.953,0.2,0.0,245,0.000,0.000,0.000,0.000,44.1
> 2017-01-01 
> 00:02:00,44.1,46,24.6,29.953,0.0,0.0,60,0.000,0.000,0.000,0.000,44.1
> 2017-01-01 
> 00:03:00,44.1,46,24.6,29.953,0.3,3.0,12,0.000,0.000,0.000,0.000,44.1
> 2017-01-01 
> 00:04:00,44.1,46,24.6,29.954,0.8,2.0,12,0.000,0.000,0.000,0.000,44.1
> 2017-01-01 
> 00:05:00,44.1,45,24.1,29.955,0.4,0.0,12,0.000,0.000,0.000,0.000,44.1
>
> I will get back to you about the WeatherCat data as soon as the import of 
> the weather-view data works.
>
> On Friday, April 21, 2017 at 7:59:20 AM UTC-4, gjr80 wrote:
>>
>> On Friday, 21 April 2017 21:35:21 UTC+10, MRL wrote:
>>>
>>> What about the date/time?
>>> The date/time is in 5 fields? Hows do I set that up in the config file?
>>>
>>  
>> If you are going to go down the path of importing from a CSV file your 
>> date-time data must be in one field. The field can be an epoch timestamp or 
>> some fixed format combination of day, month, year, hour, minute (and second 
>> if necessary) as long as it uniquely defines a moment in time. The format 
>> must be consistent throughout all rows in a given import. The way you tell 
>> wee_import about the format is through Python strptime() format codes. 
>> This is covered with examples under the heading raw_datetime_format 
>>  in the 
>> wee_import section of the Utilities Guide 
>> .
>>  
>>
>>> How could the import utility deal with the strange date/time format?
>>>
>>
>> As explained above. 
>>  
>> I would suggest you make use of the --dry-run command line option until 
>> you are happy with what you are importing. Either that or import into a 
>> test database/binding.
>>
>> My Weather-Cat data is in PDF files. Fortunately I have a program that 
>>> can convert PDF files to text files. Then a simple python program to turn 
>>> the output into CSV files.
>>>
>>
>> That's too bad, I was looking forward to making wee_import WeatherCat 
>> aware. Are the pdfs of the monthly .cat files? If so would it be possible 
>> to obtain an extract from some of the pdfs, I would prefer the raw data and 
>> am interested in the comments I believe are at the start of each file that 
>> detail the format.
>>
>> Gary
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Importing data from a CSV file

2017-04-21 Thread MRL 
I am ready to do a dry run.
It took a trivial python program to convert:
day month year hour minute temperature   humidity dewpoint   barometer 
   windspeed   gustspeed direction  rainlastmindailyrain  monthlyrain   
yearlyrain  heatindex
 1  1 2017  0  1 44.1  46 24.6 29.953 0.2 0.0 245  0.000 0.000 0.000 0.000 
44.1
 1  1 2017  0  2 44.1  46 24.6 29.953 0.0 0.0  60  0.000 0.000 0.000 0.000 
44.1
 1  1 2017  0  3 44.1  46 24.6 29.953 0.3 3.0  12  0.000 0.000 0.000 0.000 
44.1
into:
datetime,temperature,humidity,dewpoint,barometer,windspeed,gustspeed,direction,rainlastmin,dailyrain,monthlyrain,yearlyrain,heatindex
2017-01-01 
00:01:00,44.1,46,24.6,29.953,0.2,0.0,245,0.000,0.000,0.000,0.000,44.1
2017-01-01 
00:02:00,44.1,46,24.6,29.953,0.0,0.0,60,0.000,0.000,0.000,0.000,44.1
2017-01-01 
00:03:00,44.1,46,24.6,29.953,0.3,3.0,12,0.000,0.000,0.000,0.000,44.1
2017-01-01 
00:04:00,44.1,46,24.6,29.954,0.8,2.0,12,0.000,0.000,0.000,0.000,44.1
2017-01-01 
00:05:00,44.1,45,24.1,29.955,0.4,0.0,12,0.000,0.000,0.000,0.000,44.1

I will get back to you about the WeatherCat data as soon as the import of 
the weather-view data works.

On Friday, April 21, 2017 at 7:59:20 AM UTC-4, gjr80 wrote:
>
> On Friday, 21 April 2017 21:35:21 UTC+10, MRL wrote:
>>
>> What about the date/time?
>> The date/time is in 5 fields? Hows do I set that up in the config file?
>>
>  
> If you are going to go down the path of importing from a CSV file your 
> date-time data must be in one field. The field can be an epoch timestamp or 
> some fixed format combination of day, month, year, hour, minute (and second 
> if necessary) as long as it uniquely defines a moment in time. The format 
> must be consistent throughout all rows in a given import. The way you tell 
> wee_import about the format is through Python strptime() format codes. 
> This is covered with examples under the heading raw_datetime_format 
>  in the 
> wee_import section of the Utilities Guide 
> .
>  
>
>> How could the import utility deal with the strange date/time format?
>>
>
> As explained above. 
>  
> I would suggest you make use of the --dry-run command line option until 
> you are happy with what you are importing. Either that or import into a 
> test database/binding.
>
> My Weather-Cat data is in PDF files. Fortunately I have a program that can 
>> convert PDF files to text files. Then a simple python program to turn the 
>> output into CSV files.
>>
>
> That's too bad, I was looking forward to making wee_import WeatherCat 
> aware. Are the pdfs of the monthly .cat files? If so would it be possible 
> to obtain an extract from some of the pdfs, I would prefer the raw data and 
> am interested in the comments I believe are at the start of each file that 
> detail the format.
>
> Gary
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Importing data from a CSV file

2017-04-21 Thread gjr80 
On Friday, 21 April 2017 21:35:21 UTC+10, MRL wrote:
>
> What about the date/time?
> The date/time is in 5 fields? Hows do I set that up in the config file?
>
 
If you are going to go down the path of importing from a CSV file your 
date-time data must be in one field. The field can be an epoch timestamp or 
some fixed format combination of day, month, year, hour, minute (and second 
if necessary) as long as it uniquely defines a moment in time. The format 
must be consistent throughout all rows in a given import. The way you tell 
wee_import about the format is through Python strptime() format codes. This 
is covered with examples under the heading raw_datetime_format 
 in the 
wee_import section of the Utilities Guide 
.
 

> How could the import utility deal with the strange date/time format?
>

As explained above. 
 
I would suggest you make use of the --dry-run command line option until you 
are happy with what you are importing. Either that or import into a test 
database/binding.

My Weather-Cat data is in PDF files. Fortunately I have a program that can 
> convert PDF files to text files. Then a simple python program to turn the 
> output into CSV files.
>

That's too bad, I was looking forward to making wee_import WeatherCat 
aware. Are the pdfs of the monthly .cat files? If so would it be possible 
to obtain an extract from some of the pdfs, I would prefer the raw data and 
am interested in the comments I believe are at the start of each file that 
detail the format.

Gary

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Importing data from a CSV file

2017-04-20 Thread gjr80 
Don't worry about converting to csv, I would much rather have wee_import import 
from the raw files. Easiest thing is just to attach a few files, not sure if 
you will need to change the extension though, can't remember how fussy google 
groups is.

Gary

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Importing data from a CSV file

2017-04-20 Thread MRL 
Correction: The sample is from Weather-Display where I only have less than 
a year. Still working om my 3+ years of Weather-Cat.

On Thursday, April 20, 2017 at 7:24:23 PM UTC-4, MRL wrote:
>
> I have about 4 years of Weather-Cat data that I can convert to a CSV file.
> Here is the what the data looks like before conversion
>
> day month year hour minute temperature   humidity dewpoint   barometer 
>windspeed   gustspeed direction  rainlastmindailyrain  monthlyrain   
> yearlyrain  heatindex
>  1  1 2016  0  0 44.1  45 24.1 29.954 0.1 2.0 246  0.000 0.000 0.000 0.000 
> 44.1
>  1  1 2017  0  1 44.1  46 24.6 29.953 0.2 0.0 245  0.000 0.000 0.000 0.000 
> 44.1
>  1  1 2017  0  2 44.1  46 24.6 29.953 0.0 0.0  60  0.000 0.000 0.000 0.000 
> 44.1
>  1  1 2017  0  3 44.1  46 24.6 29.953 0.3 3.0  12  0.000 0.000 0.000 0.000 
> 44.1
>  1  1 2017  0  4 44.1  46 24.6 29.954 0.8 2.0  12  0.000 0.000 0.000 0.000 
> 44.1
> ...
>
> The data is by hour/minute
> I can convert the data/time data to the correct format as well as to save 
> data as a CSV file.
> I assume the first record stays and that I connect the CSV name with the 
> weewx name in the CSV config file
>
> Correct?
> Thanks
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.