Please don't take what i have written as a complaint. I appreciate all the hard work it took in the creation of WeeWx and all the contributors to the forum.
I have not looked at the content of those files it choked on to see if it has good data in it. Let me first do that. I will reply back once I have done that. Thank you. andrew On Tuesday, September 1, 2020 at 5:20:30 PM UTC-4 gjr80 wrote: > If you to provide some details on problems the handful of files > experienced I am happy to look at wee_import to see if any changes can be > made to improve its handling of such files. The WD import module of > wee_import was developed based on a handful of WD log files found on the > internet, so it is quite possible there are some corner cases that may > cause wee_import to reject a file. > > Gary > > On Wednesday, 2 September 2020 at 07:08:20 UTC+10 Andrew M wrote: > >> After many, many hours the wee-import processed all but a handful >> of files. Have several that choked on the import. >> Ended up with 3,873,678 records. >> >> Next I need to validate that the data imported to look for any potential >> bad data. >> >> >> >> On Friday, August 21, 2020 at 7:31:36 PM UTC-4 Andrew M wrote: >> >>> Thank you for the response. >>> >>> I actually changed up how I am going about this. My WD ran on a >>> Windows10 box. I had a RaspPi box sitting in a box so i decided to use that >>> for WeeWx. >>> I was formatting the hard drive on the Win10 machine when i had the >>> thought that I should just put Debian on that and use that for WeeWx. >>> Which I did. This machine is a little faster that the RaspPi , so one I >>> have other things straight I will use that for the conversion. Migrating >>> the MySql from WD to the tables that WeeWX has is probably doable, but >>> really don't want to sit down and think about it that much. I will just let >>> the current weather collect on the RaspPi box and then once the import is >>> done on the now Debian box I can combine the two DB much easier. >>> >>> I have 60k rows on the MySQL DB with my current webhost, but more than >>> that saved external to that on a hard drive from a previous webhost that I >>> never moved. >>> >>> >>> >>> On Wednesday, August 19, 2020 at 8:37:29 PM UTC-4 gjr80 wrote: >>> >>>> Sorry can't help you with MySQL to MySQL migration. >>>> >>>> Regards wee_import though, yes it can be slow. When I wrote the WD >>>> import module the WD user who first used it in anger had something like >>>> (from memory) 10 years of data to import. The import had to be done in >>>> batches (again from memory) of 2-3 years each (wee_import uses >>>> transactions on the database but does keep track of duplicate timestamps >>>> and a few other things so memory usage does grow as the span of the import >>>> grows). I found one email from the user with the results of the first >>>> batch >>>> import and 1.4 millions records were imported on a Raspberry Pi in 58 >>>> minutes. Al things considered I find that reasonable. >>>> >>>> There are a couple of things you can do to speed up wee_import. You >>>> can tweak a the tranche setting in the import config file, this alters >>>> the size of the transactions (in records) that wee_import uses. The >>>> default >>>> is 250, you could raise this which will result in fewer db transactions >>>> but >>>> it will likely increase memory usage so you may need to do the import in >>>> smaller batches. One other approach if using a slow(ish) RPi as your WeeWX >>>> machine is to do just the import on a faster machine and then copy the >>>> imported data to the WeeWX RPi. Granted this is simpler when using SQLite >>>> but depending on your setup could be adapted for MySQL. >>>> >>>> You say you have 60 000 odd MySQL records, that does not seem like >>>> much, how does that correlate with the number of entries in the WD log >>>> files? >>>> >>>> Gary >>>> >>>> On Thursday, 20 August 2020 08:48:46 UTC+10, Andrew M wrote: >>>>> >>>>> I started to use the wee_import process to process all my WD log files >>>>> to WeeWx MySqlDB, and it was taking a long time. It seems like If I have >>>>> many years of data it will take that long to import them into WeeWx >>>>> MySqlDB. >>>>> >>>>> I then thought, oh wait, i already have the WD data in a MySql DB, so >>>>> why am i doing this process. >>>>> >>>>> Now I have to figure out how I can gracefully import all the WD data I >>>>> have in a MySQL DB to the one I set up for WeeWx. Both are on the same >>>>> hosted server. Different DB names. >>>>> >>>>> There is one table for WD and multiple tables for WeeWx so have no >>>>> idea on where to begin with this. I have ~60,000 rows of data in the WD >>>>> MySQL DB table. >>>>> >>>>> Does anyone have a graceful way of migrating the WD MySQL DB into a >>>>> WeeWX DB? >>>>> >>>>> Am I overlooking something in the documentation or in group? >>>>> >>>>> Thank you. >>>>> >>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/927d9411-d6e0-466a-a498-c983debd5f18n%40googlegroups.com.
