OK, working on this some more... strlcpy isn't available on Windows (and appears not to be there by defauly on all Linux flavors that I have access to, either... some have it in bsd/string.h, some don't). So let's use the universally available strncpy instead.
I removed all the "glerch" markings on your comments. I changed some of the more annoying variable names but more fixing is needed. I broke things down into a lot of preparatory patches and the remaining big logic change. That last change hasn't received a lot of review and love, yet. I'm running out of time but I wanted to send out what I have so that others can comment and so that you, Guido, can take a look at what I did and how I structured things. /D
>From 3a074b61d1a0f5ed66d81c5f4f38ab52f098ea47 Mon Sep 17 00:00:00 2001 From: Guido Lerch <[email protected]> Date: Sat, 5 Sep 2015 10:29:43 -0700 Subject: [PATCH 01/10] Uemis downloader: small whitespace change This makes the comments line up more nicely and fixes some other random issues in preparation for the following changes. [Dirk Hohndel: refactored one huge commit into smaller pieces] Signed-off-by: Guido Lerch <[email protected]> Signed-off-by: Dirk Hohndel <[email protected]> --- uemis-downloader.c | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/uemis-downloader.c b/uemis-downloader.c index baf9484154e2..630eacc64793 100644 --- a/uemis-downloader.c +++ b/uemis-downloader.c @@ -9,6 +9,8 @@ * I believe that I only used the information about HOW to do this (download data from the Uemis * Zurich) but did not actually use any of his copyrighted code, therefore the license under which * he released his code does not apply to this new implementation in C + * + * Modified by Guido Lerch [email protected] in August 2015 */ #include <fcntl.h> #include <dirent.h> @@ -28,14 +30,15 @@ #define BUFLEN 2048 #define NUM_PARAM_BUFS 10 -#if UEMIS_DEBUG & 64 /* we are reading from a copy of the filesystem, not the device - no need to wait */ -#define UEMIS_TIMEOUT 50 /* 50ns */ -#define UEMIS_LONG_TIMEOUT 500 /* 500ns */ -#define UEMIS_MAX_TIMEOUT 2000 /* 2ms */ + +#if UEMIS_DEBUG & 64 /* we are reading from a copy of the filesystem, not the device - no need to wait */ +#define UEMIS_TIMEOUT 50 /* 50ns */ +#define UEMIS_LONG_TIMEOUT 500 /* 500ns */ +#define UEMIS_MAX_TIMEOUT 2000 /* 2ms */ #else -#define UEMIS_TIMEOUT 50000 /* 50ms */ -#define UEMIS_LONG_TIMEOUT 500000 /* 500ms */ -#define UEMIS_MAX_TIMEOUT 2000000 /* 2s */ +#define UEMIS_TIMEOUT 50000 /* 50ms */ +#define UEMIS_LONG_TIMEOUT 500000 /* 500ms */ +#define UEMIS_MAX_TIMEOUT 2000000 /* 2s */ #endif #ifdef UEMIS_DEBUG @@ -107,8 +110,8 @@ static void uemis_add_string(const char *buffer, char **text) static void uemis_get_weight(char *buffer, weightsystem_t *weight, int diveid) { weight->weight.grams = uemis_get_weight_unit(diveid) ? - lbs_to_grams(ascii_strtod(buffer, NULL)) : - ascii_strtod(buffer, NULL) * 1000; + lbs_to_grams(ascii_strtod(buffer, NULL)) : + ascii_strtod(buffer, NULL) * 1000; weight->description = strdup(translate("gettextFromC", "unknown")); } @@ -615,7 +618,7 @@ static bool uemis_get_answer(const char *path, char *request, int n_param_in, #endif return found_answer; fs_error: - close (ans_file); + close(ans_file); return false; } @@ -666,9 +669,9 @@ static void track_divespot(char *val, int diveid, uint32_t dive_site_uuid) return; } -static char *suit[] = { "", QT_TRANSLATE_NOOP("gettextFromC", "wetsuit"), QT_TRANSLATE_NOOP("gettextFromC", "semidry"), QT_TRANSLATE_NOOP("gettextFromC", "drysuit") }; -static char *suit_type[] = { "", QT_TRANSLATE_NOOP("gettextFromC", "shorty"), QT_TRANSLATE_NOOP("gettextFromC", "vest"), QT_TRANSLATE_NOOP("gettextFromC", "long john"), QT_TRANSLATE_NOOP("gettextFromC", "jacket"), QT_TRANSLATE_NOOP("gettextFromC", "full suit"), QT_TRANSLATE_NOOP("gettextFromC", "2 pcs full suit") }; -static char *suit_thickness[] = { "", "0.5-2mm", "2-3mm", "3-5mm", "5-7mm", "8mm+", QT_TRANSLATE_NOOP("gettextFromC", "membrane") }; +static char *suit[] = {"", QT_TRANSLATE_NOOP("gettextFromC", "wetsuit"), QT_TRANSLATE_NOOP("gettextFromC", "semidry"), QT_TRANSLATE_NOOP("gettextFromC", "drysuit")}; +static char *suit_type[] = {"", QT_TRANSLATE_NOOP("gettextFromC", "shorty"), QT_TRANSLATE_NOOP("gettextFromC", "vest"), QT_TRANSLATE_NOOP("gettextFromC", "long john"), QT_TRANSLATE_NOOP("gettextFromC", "jacket"), QT_TRANSLATE_NOOP("gettextFromC", "full suit"), QT_TRANSLATE_NOOP("gettextFromC", "2 pcs full suit")}; +static char *suit_thickness[] = {"", "0.5-2mm", "2-3mm", "3-5mm", "5-7mm", "8mm+", QT_TRANSLATE_NOOP("gettextFromC", "membrane")}; static void parse_tag(struct dive *dive, char *tag, char *val) { -- 2.5.1
>From a547ee0ff2044637d2cf300655b509616b49bac1 Mon Sep 17 00:00:00 2001 From: Guido Lerch <[email protected]> Date: Sat, 5 Sep 2015 10:33:43 -0700 Subject: [PATCH 02/10] Uemis downloader: various changes to the debugging output This shouldn't change any of the actualy code, except when it comes to debugging output. [Dirk Hohndel: refactored one huge commit into smaller pieces] Signed-off-by: Guido Lerch <[email protected]> Signed-off-by: Dirk Hohndel <[email protected]> --- uemis-downloader.c | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/uemis-downloader.c b/uemis-downloader.c index 630eacc64793..8c912d4cfebe 100644 --- a/uemis-downloader.c +++ b/uemis-downloader.c @@ -30,6 +30,8 @@ #define BUFLEN 2048 #define NUM_PARAM_BUFS 10 +// debugging setup +//#define UEMIS_DEBUG 1 + 2 #if UEMIS_DEBUG & 64 /* we are reading from a copy of the filesystem, not the device - no need to wait */ #define UEMIS_TIMEOUT 50 /* 50ns */ @@ -375,7 +377,7 @@ static char *first_object_id_val(char *buf) char *p = object + 14; char *t = tmp; -#if UEMIS_DEBUG & 2 +#if UEMIS_DEBUG & 4 char debugbuf[50]; strncpy(debugbuf, object, 49); debugbuf[49] = '\0'; @@ -411,8 +413,8 @@ static void show_progress(char *buf, const char *what) char *val = first_object_id_val(buf); if (val) { /* let the user know what we are working on */ -#if UEMIS_DEBUG & 2 - fprintf(stderr, "reading %s %s %s\n", what, val, buf); +#if UEMIS_DEBUG & 16 + fprintf(debugfile, "reading %s\n %s\ %s\n", what, val, buf); #endif uemis_info(translate("gettextFromC", "%s %s"), what, val); free(val); @@ -448,7 +450,9 @@ static bool uemis_get_answer(const char *path, char *request, int n_param_in, reqtxt_file = subsurface_open(reqtxt_path, O_RDWR | O_CREAT, 0666); if (reqtxt_file == -1) { *error_text = "can't open req.txt"; - fprintf(stderr, "open %s failed with errno %d\n", reqtxt_path, errno); +#ifdef UEMIS_DEBUG + fprintf(debugfile, "open %s failed with errno %d\n", reqtxt_path, errno); +#endif return false; } snprintf(sb, BUFLEN, "n%04d12345678", filenr); @@ -470,7 +474,7 @@ static bool uemis_get_answer(const char *path, char *request, int n_param_in, file_length = strlen(sb); snprintf(fl, 10, "%08d", file_length - 13); memcpy(sb + 5, fl, strlen(fl)); -#if UEMIS_DEBUG & 1 +#if UEMIS_DEBUG & 4 fprintf(debugfile, "::w req.txt \"%s\"\n", sb); #endif if (write(reqtxt_file, sb, strlen(sb)) != strlen(sb)) { @@ -677,6 +681,10 @@ static void parse_tag(struct dive *dive, char *tag, char *val) { /* we can ignore computer_id, water and gas as those are redundant * with the binary data and would just get overwritten */ +#if UEMIS_DEBUG & 4 + if (strcmp(tag, "file_content")) + fprintf(debugfile, "Adding to dive %d : %s = %s\n", dive->dc.diveid, tag, val); +#endif if (!strcmp(tag, "date")) { uemis_ts(val, &dive->when); } else if (!strcmp(tag, "duration")) { @@ -760,7 +768,7 @@ static bool process_raw_buffer(device_data_t *devdata, uint32_t deviceid, char * /* remember, we don't know if this is the right entry, * so first test if this is even a valid entry */ if (strstr(inbuf, "deleted{bool{true")) { -#if UEMIS_DEBUG & 4 +#if UEMIS_DEBUG & 2 fprintf(debugfile, "p_r_b entry deleted\n"); #endif /* oops, this one isn't valid, suggest to try the previous one */ @@ -795,9 +803,16 @@ static bool process_raw_buffer(device_data_t *devdata, uint32_t deviceid, char * } val = next_token(&bp); if (log && !strcmp(tag, "object_id")) { +#if UEMIS_DEBUG & 8 + if (strlen(val) < 20) + fprintf(debugfile, "Parsed %s, %s, %s\n*************************\n", tag, type, val); +#endif free(*max_divenr); *max_divenr = strdup(val); dive->dc.diveid = atoi(val); +#if UEMIS_DEBUG % 2 + fprintf(debugfile, "Adding new dive from log with object_id %d.\n", atoi(val)); +#endif if (keep_number) dive->number = atoi(val); } else if (!log && !strcmp(tag, "logfilenr")) { @@ -808,6 +823,9 @@ static bool process_raw_buffer(device_data_t *devdata, uint32_t deviceid, char * } else if (!log && dive && !strcmp(tag, "divespot_id")) { dive->dive_site_uuid = create_dive_site("from Uemis", dive->when); track_divespot(val, dive->dc.diveid, dive->dive_site_uuid); +#if UEMIS_DEBUG & 2 + fprintf(debugfile, "Created divesite %d for diveid : %d\n", dive->dive_site_uuid, dive->dc.diveid); +#endif } else if (dive) { parse_tag(dive, tag, val); } -- 2.5.1
>From 8647f959534396e1ee6d0685908ba9801afb646b Mon Sep 17 00:00:00 2001 From: Guido Lerch <[email protected]> Date: Sat, 5 Sep 2015 11:00:54 -0700 Subject: [PATCH 03/10] Uemis downloader: fix and move helper function This function is only used in the Uemis downloader, and it got broken when we switched to using a separate table for the downloaded dives. Signed-off-by: Dirk Hohndel <[email protected]> --- dive.h | 15 --------------- uemis-downloader.c | 11 ++++++++++- 2 files changed, 10 insertions(+), 16 deletions(-) diff --git a/dive.h b/dive.h index eac6e724d08d..c20b59bd377c 100644 --- a/dive.h +++ b/dive.h @@ -582,21 +582,6 @@ extern void delete_current_divecomputer(void); #define for_each_gps_location(_i, _x) \ for ((_i) = 0; ((_x) = get_gps_location(_i, &gps_location_table)) != NULL; (_i)++) -static inline struct dive *get_dive_by_uemis_diveid(uint32_t diveid, uint32_t deviceid) -{ - int i; - struct dive *dive; - - for_each_dive (i, dive) { - struct divecomputer *dc = &dive->dc; - do { - if (dc->diveid == diveid && dc->deviceid == deviceid) - return dive; - } while ((dc = dc->next) != NULL); - } - return NULL; -} - static inline struct dive *get_dive_by_uniq_id(int id) { int i; diff --git a/uemis-downloader.c b/uemis-downloader.c index 8c912d4cfebe..340daee609e9 100644 --- a/uemis-downloader.c +++ b/uemis-downloader.c @@ -126,6 +126,15 @@ static struct dive *uemis_start_dive(uint32_t deviceid) return dive; } +static struct dive *get_dive_by_uemis_diveid(device_data_t *devdata, u_int32_t object_id) +{ + for (int i = 0; i < devdata->download_table->nr; i++) { + if (object_id == devdata->download_table->dives[i]->dc.diveid) + return devdata->download_table->dives[i]; + } + return NULL; +} + static void record_uemis_dive(device_data_t *devdata, struct dive *dive) { if (devdata->create_new_trip) { @@ -817,7 +826,7 @@ static bool process_raw_buffer(device_data_t *devdata, uint32_t deviceid, char * dive->number = atoi(val); } else if (!log && !strcmp(tag, "logfilenr")) { /* this one tells us which dive we are adding data to */ - dive = get_dive_by_uemis_diveid(atoi(val), deviceid); + dive = get_dive_by_uemis_diveid(devdata, atoi(val)); if (for_dive) *for_dive = atoi(val); } else if (!log && dive && !strcmp(tag, "divespot_id")) { -- 2.5.1
>From b7f5871e597d8b0088d75ef858be10cc93e27ad3 Mon Sep 17 00:00:00 2001 From: Guido Lerch <[email protected]> Date: Sat, 5 Sep 2015 11:04:58 -0700 Subject: [PATCH 04/10] Uemis downloader: add constant for maximum number of ANS files [Dirk Hohndel: refactored one huge commit into smaller pieces] Signed-off-by: Guido Lerch <[email protected]> Signed-off-by: Dirk Hohndel <[email protected]> --- uemis-downloader.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/uemis-downloader.c b/uemis-downloader.c index 340daee609e9..b263072ee0b9 100644 --- a/uemis-downloader.c +++ b/uemis-downloader.c @@ -33,6 +33,8 @@ // debugging setup //#define UEMIS_DEBUG 1 + 2 +#define UEMIS_MAX_FILES 4000 + #if UEMIS_DEBUG & 64 /* we are reading from a copy of the filesystem, not the device - no need to wait */ #define UEMIS_TIMEOUT 50 /* 50ns */ #define UEMIS_LONG_TIMEOUT 500 /* 500ns */ @@ -501,7 +503,7 @@ static bool uemis_get_answer(const char *path, char *request, int n_param_in, while (searching || assembling_mbuf) { if (import_thread_cancelled) return false; - progress_bar_fraction = filenr / 4000.0; + progress_bar_fraction = filenr / (double)UEMIS_MAX_FILES; snprintf(fl, 13, "ANS%d.TXT", filenr - 1); ans_path = build_filename(build_filename(path, "ANS"), fl); ans_file = subsurface_open(ans_path, O_RDONLY, 0666); -- 2.5.1
>From 4ba5dd85d4078d320b4d16d94fb3f973e54fcadb Mon Sep 17 00:00:00 2001 From: Guido Lerch <[email protected]> Date: Sat, 5 Sep 2015 11:09:10 -0700 Subject: [PATCH 05/10] Uemis downloader: parse_divespot should return if it was successful But sadly the code then doesn't actually use that return value, so this is not quite perfect, yet. [Dirk Hohndel: refactored one huge commit into smaller pieces] Signed-off-by: Guido Lerch <[email protected]> Signed-off-by: Dirk Hohndel <[email protected]> --- uemis-downloader.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/uemis-downloader.c b/uemis-downloader.c index b263072ee0b9..8c72354ef8e0 100644 --- a/uemis-downloader.c +++ b/uemis-downloader.c @@ -637,7 +637,7 @@ fs_error: return false; } -static void parse_divespot(char *buf) +static bool parse_divespot(char *buf) { char *bp = buf + 1; char *tp = next_token(&bp); @@ -646,14 +646,17 @@ static void parse_divespot(char *buf) int divespot, len; double latitude = 0.0, longitude = 0.0; - + // dive spot got deleted, so fail here + if (strstr(bp, "deleted{bool{true")) + return false; + // not a dive spot, fail here if (strcmp(tp, "divespot")) - return; + return false; do tag = next_token(&bp); while (*tag && strcmp(tag, "object_id")); if (!*tag) - return; + return false; next_token(&bp); val = next_token(&bp); divespot = atoi(val); @@ -672,7 +675,9 @@ static void parse_divespot(char *buf) latitude = ascii_strtod(val, NULL); } } while (tag && *tag); + uemis_set_divelocation(divespot, locationstring, longitude, latitude); + return true; } static void track_divespot(char *val, int diveid, uint32_t dive_site_uuid) -- 2.5.1
>From ff0859ff213bca011e33c74db01e1e8e4803fa18 Mon Sep 17 00:00:00 2001 From: Guido Lerch <[email protected]> Date: Sat, 5 Sep 2015 11:15:05 -0700 Subject: [PATCH 06/10] Uemis downloader: add helper function to estimate memory needs [Dirk Hohndel: refactored one huge commit into smaller pieces] Signed-off-by: Guido Lerch <[email protected]> Signed-off-by: Dirk Hohndel <[email protected]> --- uemis-downloader.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/uemis-downloader.c b/uemis-downloader.c index 8c72354ef8e0..16d7811aec41 100644 --- a/uemis-downloader.c +++ b/uemis-downloader.c @@ -34,6 +34,9 @@ //#define UEMIS_DEBUG 1 + 2 #define UEMIS_MAX_FILES 4000 +#define UEMIS_MEM_FULL 3 +#define UEMIS_MEM_CRITICAL 1 +#define UEMIS_MEM_OK 0 #if UEMIS_DEBUG & 64 /* we are reading from a copy of the filesystem, not the device - no need to wait */ #define UEMIS_TIMEOUT 50 /* 50ns */ @@ -900,6 +903,39 @@ static char *uemis_get_divenr(char *deviceidstr) return strdup(divenr); } +/* do some more sophisticated calculations here to try and predict if the next round of + * divelog/divedetail reads will fit into the UEMIS buffer, + * filenr holds now the uemis filenr after having read several logs including the dive details, + * fCapacity will five us the average number of files needed for all currently loaded data + * remember the maximum file usage per dive + * return : UEMIS_MEM_OK if there is enough memeory for a full round + * UEMIS_MEM_CRITICAL if the memory is good for reading the dive logs + * UEMIS_MEM_FULL if the memory is exhaused + */ +static int get_memory(struct dive_table *td) +{ + + if (td->nr == 0) + return UEMIS_MEM_OK; + + if (filenr / td->nr > max_mem_used) + max_mem_used = filenr / td->nr; + /* predict based on the max_mem_used value if the set of next 11 divelogs plus details + * fit into the memory before we have to disconnect the UEMIS and continuem. To be on + * the safe side we calculate using 12 dives. */ + if (max_mem_used * 11 > UEMIS_MAX_FILES - filenr) { + /* the next set of divelogs will most likely not fit into the memory */ + if (nr_divespots * 2 > UEMIS_MAX_FILES - filenr) { + /* if we get here we have a severe issue as the divespots will not fit into + * this run either. */ + return UEMIS_MEM_FULL; + } + /* we continue reading the divespots */ + return UEMIS_MEM_CRITICAL; + } + return UEMIS_MEM_OK; +} + const char *do_uemis_import(device_data_t *data) { const char *mountpath = data->devname; -- 2.5.1
>From 040caaa00b6874e1e93320d4c1c1aa6abba8fee4 Mon Sep 17 00:00:00 2001 From: Guido Lerch <[email protected]> Date: Sat, 5 Sep 2015 11:20:13 -0700 Subject: [PATCH 07/10] Uemis downloader: parse dive_no as well [Dirk Hohndel: refactored one huge commit into smaller pieces] Signed-off-by: Guido Lerch <[email protected]> Signed-off-by: Dirk Hohndel <[email protected]> --- uemis-downloader.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/uemis-downloader.c b/uemis-downloader.c index 16d7811aec41..fdaea876abce 100644 --- a/uemis-downloader.c +++ b/uemis-downloader.c @@ -727,6 +727,8 @@ static void parse_tag(struct dive *dive, char *tag, char *val) } else if (!strcmp(tag, "u8SuitThickness")) { if (*suit_thickness[atoi(val)]) uemis_add_string(translate("gettextFromC", suit_thickness[atoi(val)]), &dive->suit); + } else if (!strcmp(tag, "dive_no")) { + dive->number = atoi(val); } } -- 2.5.1
>From fa56d5d1361b66bc453009f32dfe4b1b90ddca32 Mon Sep 17 00:00:00 2001 From: Guido Lerch <[email protected]> Date: Sat, 5 Sep 2015 11:21:14 -0700 Subject: [PATCH 08/10] Uemis downloader: add heler function to delete dive [Dirk Hohndel: refactored one huge commit into smaller pieces] Signed-off-by: Guido Lerch <[email protected]> Signed-off-by: Dirk Hohndel <[email protected]> --- uemis-downloader.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/uemis-downloader.c b/uemis-downloader.c index fdaea876abce..a503f91a776a 100644 --- a/uemis-downloader.c +++ b/uemis-downloader.c @@ -732,6 +732,40 @@ static void parse_tag(struct dive *dive, char *tag, char *val) } } +static bool uemis_delete_dive(device_data_t *devdata, uint32_t diveid) +{ + struct dive *dive = NULL; + + if (devdata->download_table->dives[devdata->download_table->nr - 1]->dc.diveid == diveid) { + /* we hit the last one in the array */ + dive = devdata->download_table->dives[devdata->download_table->nr - 1]; + } else { + for (int i = 0; i < devdata->download_table->nr - 1; i++) { + if (devdata->download_table->dives[i]->dc.diveid == diveid) { + dive = devdata->download_table->dives[i]; + for (int x = i; x < devdata->download_table->nr - 1; x++) + devdata->download_table->dives[i] = devdata->download_table->dives[x + 1]; + } + } + } + if (dive) { + devdata->download_table->dives[--devdata->download_table->nr] = NULL; + if (dive->tripflag != TF_NONE) + remove_dive_from_trip(dive, false); + + free(dive->dc.sample); + free((void *)dive->notes); + free((void *)dive->divemaster); + free((void *)dive->buddy); + free((void *)dive->suit); + taglist_free(dive->tag_list); + free(dive); + + return true; + } + return false; +} + /* This function is called for both divelog and dive information that we get * from the SDA (what an insane design, btw). The object_id in the divelog * matches the logfilenr in the dive information (which has its own, often -- 2.5.1
>From 9d5fe3af6088578f4c816fae04c8bdd02dd95c3a Mon Sep 17 00:00:00 2001 From: Guido Lerch <[email protected]> Date: Sat, 5 Sep 2015 11:22:22 -0700 Subject: [PATCH 09/10] Uemis downloader: add helper function to dump buffer [Dirk Hohndel: refactored one huge commit into smaller pieces] Signed-off-by: Guido Lerch <[email protected]> Signed-off-by: Dirk Hohndel <[email protected]> --- uemis-downloader.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/uemis-downloader.c b/uemis-downloader.c index a503f91a776a..6f7140640a6f 100644 --- a/uemis-downloader.c +++ b/uemis-downloader.c @@ -939,6 +939,42 @@ static char *uemis_get_divenr(char *deviceidstr) return strdup(divenr); } +static int bufCnt = 0; +static bool do_dump_buffer_to_file(char *buf, char *prefix, int round) +{ + char path[100]; + char date[40]; + char obid[40]; + if (!buf) + return false; + + if (strstr(buf, "date{ts{")) + strncpy(date, strstr(buf, "date{ts{"), sizeof(date)); + else + strncpy(date, strdup("date{ts{no-date{"), sizeof(date)); + + if (!strstr(buf, "object_id{int{")) + return false; + + strncpy(obid, strstr(buf, "object_id{int{"), sizeof(obid)); + char *ptr1 = strstr(date, "date{ts{"); + char *ptr2 = strstr(obid, "object_id{int{"); + char *pdate = next_token(&ptr1); + pdate = next_token(&ptr1); + pdate = next_token(&ptr1); + char *pobid = next_token(&ptr2); + pobid = next_token(&ptr2); + pobid = next_token(&ptr2); + snprintf(path, sizeof(path), "/Users/glerch/UEMIS Dump/%s_%s_Uemis_dump_%s_in_round_%d_%d.txt", prefix, pdate, pobid, round, bufCnt); + int dumpFile = subsurface_open(path, O_RDWR | O_CREAT, 0666); + if (dumpFile == -1) + return false; + write(dumpFile, buf, strlen(buf)); + close(dumpFile); + bufCnt++; + return true; +} + /* do some more sophisticated calculations here to try and predict if the next round of * divelog/divedetail reads will fit into the UEMIS buffer, * filenr holds now the uemis filenr after having read several logs including the dive details, -- 2.5.1
>From cd623b484c9a03b5213674a8bd33540ee950df4a Mon Sep 17 00:00:00 2001 From: Guido Lerch <[email protected]> Date: Sat, 5 Sep 2015 11:24:10 -0700 Subject: [PATCH 10/10] Uemis downloader: change download logic - change the way how dive logs are mapped to dive details in do_uemis_import() - dives deleted on the Uemis will not be downloaded anymore (added function uemis_delete_dive_by_diveid) - change the way the memory consumption was checked to acknowledge very large files (trying to predic on how many more files fit into the buffer by calculating an average consumbtion over each divelogs block) - minimal design change to support the above [Dirk Hohndel: refactored one huge commit into smaller pieces] Signed-off-by: Guido Lerch <[email protected]> Signed-off-by: Dirk Hohndel <[email protected]> --- uemis-downloader.c | 314 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 232 insertions(+), 82 deletions(-) diff --git a/uemis-downloader.c b/uemis-downloader.c index 6f7140640a6f..2b8cd6c71f3b 100644 --- a/uemis-downloader.c +++ b/uemis-downloader.c @@ -59,9 +59,13 @@ static int filenr; static int number_of_files; static char *mbuf = NULL; static int mbuf_size = 0; - static int nr_divespots = -1; +static int buddies_start = 0; +static int buddies = -1; +static int max_mem_used = -1; +static int next_table_index = 0; + /* helper function to parse the Uemis data structures */ static void uemis_ts(char *buffer, void *_when) { @@ -784,7 +788,7 @@ static bool process_raw_buffer(device_data_t *devdata, uint32_t deviceid, char * bool done = false; int inbuflen = strlen(inbuf); char *endptr = buf + inbuflen; - bool log = false; + bool isLog, isDive = false; char *sections[10]; int s, nr_sections = 0; struct dive *dive = NULL; @@ -798,7 +802,7 @@ static bool process_raw_buffer(device_data_t *devdata, uint32_t deviceid, char * tp = next_token(&bp); if (strcmp(tp, "divelog") == 0) { /* this is a divelog */ - log = true; + isLog = true; tp = next_token(&bp); /* is it a valid entry or nothing ? */ if (strcmp(tp, "1.0") != 0 || strstr(inbuf, "divelog{1.0{{{{")) { @@ -807,6 +811,7 @@ static bool process_raw_buffer(device_data_t *devdata, uint32_t deviceid, char * } } else if (strcmp(tp, "dive") == 0) { /* this is dive detail */ + isDive = true; tp = next_token(&bp); if (strcmp(tp, "1.0") != 0) { free(buf); @@ -817,7 +822,7 @@ static bool process_raw_buffer(device_data_t *devdata, uint32_t deviceid, char * free(buf); return false; } - if (log) { + if (isLog) { dive = uemis_start_dive(deviceid); } else { /* remember, we don't know if this is the right entry, @@ -857,26 +862,30 @@ static bool process_raw_buffer(device_data_t *devdata, uint32_t deviceid, char * continue; } val = next_token(&bp); - if (log && !strcmp(tag, "object_id")) { #if UEMIS_DEBUG & 8 if (strlen(val) < 20) fprintf(debugfile, "Parsed %s, %s, %s\n*************************\n", tag, type, val); #endif + if (isLog && strcmp(tag, "object_id") == 0) { free(*max_divenr); *max_divenr = strdup(val); dive->dc.diveid = atoi(val); #if UEMIS_DEBUG % 2 fprintf(debugfile, "Adding new dive from log with object_id %d.\n", atoi(val)); #endif + /* glerch Sep. 2015 + * maybe I am missing something here but this seems wrong if (keep_number) dive->number = atoi(val); - } else if (!log && !strcmp(tag, "logfilenr")) { + */ + } else if (isDive && strcmp(tag, "logfilenr") == 0) { /* this one tells us which dive we are adding data to */ dive = get_dive_by_uemis_diveid(devdata, atoi(val)); if (for_dive) *for_dive = atoi(val); - } else if (!log && dive && !strcmp(tag, "divespot_id")) { - dive->dive_site_uuid = create_dive_site("from Uemis", dive->when); + } else if (!isLog && dive && !strcmp(tag, "divespot_id")) { + timestamp_t t; + dive->dive_site_uuid = create_dive_site("from Uemis", (int)time(NULL)); track_divespot(val, dive->dc.diveid, dive->dive_site_uuid); #if UEMIS_DEBUG & 2 fprintf(debugfile, "Created divesite %d for diveid : %d\n", dive->dive_site_uuid, dive->dc.diveid); @@ -884,7 +893,7 @@ static bool process_raw_buffer(device_data_t *devdata, uint32_t deviceid, char * } else if (dive) { parse_tag(dive, tag, val); } - if (log && !strcmp(tag, "file_content")) + if (isLog && !strcmp(tag, "file_content")) done = true; /* done with one dive (got the file_content tag), but there could be more: * a '{' indicates the end of the record - but we need to see another "{{" @@ -897,7 +906,7 @@ static bool process_raw_buffer(device_data_t *devdata, uint32_t deviceid, char * dive = uemis_start_dive(deviceid); } } - if (log) { + if (isLog) { if (dive->dc.diveid) { record_uemis_dive(devdata, dive); mark_divelist_changed(true); @@ -928,7 +937,7 @@ static char *uemis_get_divenr(char *deviceidstr) struct dive *d; for_each_dive (i, d) { struct divecomputer *dc; - for_each_dc(d, dc) { + for_each_dc (d, dc) { if (dc->model && !strcmp(dc->model, "Uemis Zurich") && (dc->deviceid == 0 || dc->deviceid == 0x7fffffff || dc->deviceid == deviceid) && dc->diveid > maxdiveid) @@ -1008,43 +1017,65 @@ static int get_memory(struct dive_table *td) return UEMIS_MEM_OK; } +/* mark a dive as deleted by setting download to false + * this will be picked up by some cleaning statement later */ +static void do_delete_dives(struct dive_table *td, int idx) +{ + for (int x = idx; x < td->nr; x++) + td->dives[x]->downloaded = false; +} + const char *do_uemis_import(device_data_t *data) { const char *mountpath = data->devname; short force_download = data->force_download; char *newmax = NULL; int first, start, end = -2; - int i, offset = 0; + int i = 0; uint32_t deviceidnr; - char objectid[10]; + //char objectid[10]; char *deviceid = NULL; const char *result = NULL; char *endptr; bool success, keep_number = false, once = true; + char divetoRead[10]; + char logFileNoToFind[20]; + int deletedFiles = 0; + int lastFoundLogfilenr = 0; + int iMatchDiveAndLog = 0; + int iStartCleanup = 0; + struct dive_table *td = NULL; + struct dive *dive = NULL; + int uemis_mem_status = UEMIS_MEM_OK; + + const char *dTime; if (dive_table.nr == 0) keep_number = true; + uemis_info(translate("gettextFromC", "Initialise communication")); if (!uemis_init(mountpath)) { free(reqtxt_path); return translate("gettextFromC", "Uemis init failed"); } + if (!uemis_get_answer(mountpath, "getDeviceId", 0, 1, &result)) goto bail; deviceid = strdup(param_buff[0]); deviceidnr = atoi(deviceid); - /* the answer from the DeviceId call becomes the input parameter for getDeviceData */ - if (!uemis_get_answer(mountpath, "getDeviceData", 1, 0, &result)) - goto bail; + /* param_buff[0] is still valid */ if (!uemis_get_answer(mountpath, "initSession", 1, 6, &result)) goto bail; + uemis_info(translate("gettextFromC", "Start download")); if (!uemis_get_answer(mountpath, "processSync", 0, 2, &result)) goto bail; + /* before starting the long download, check if user pressed cancel */ if (import_thread_cancelled) goto bail; + param_buff[1] = "notempty"; /* if we force it we start downloading from the first dive on the Uemis; * otherwise check which was the last dive downloaded */ @@ -1052,108 +1083,227 @@ const char *do_uemis_import(device_data_t *data) newmax = uemis_get_divenr(deviceid); else newmax = strdup("0"); + + // newmax = strdup("240"); first = start = atoi(newmax); + +#if UEMIS_DEBUG & 2 + int rounds = -1; + int round = 0; +#endif for (;;) { +#if UEMIS_DEBUG & 2 + round++; +#endif #if UEMIS_DEBUG & 4 fprintf(debugfile, "d_u_i inner loop start %d end %d newmax %s\n", start, end, newmax); #endif + /* start at the last filled download table index */ + iStartCleanup = iMatchDiveAndLog = data->download_table->nr; + sprintf(newmax, "%d", start); param_buff[2] = newmax; param_buff[3] = 0; success = uemis_get_answer(mountpath, "getDivelogs", 3, 0, &result); - /* process the buffer we have assembled */ - if (mbuf) + uemis_mem_status = get_memory(data->download_table); + if (success && mbuf && uemis_mem_status != UEMIS_MEM_FULL) { +#if UEMIS_DEBUG % 2 + do_dump_buffer_to_file(mbuf, strdup("Divelogs"), round); +#endif + /* process the buffer we have assembled */ + if (!process_raw_buffer(data, deviceidnr, mbuf, &newmax, keep_number, NULL)) { /* if no dives were downloaded, mark end appropriately */ if (end == -2) end = start - 1; success = false; } - if (once) { - char *t = first_object_id_val(mbuf); - if (t && atoi(t) > start) - start = atoi(t); - free(t); - once = false; - } - /* clean up mbuf */ - endptr = strstr(mbuf, "{{{"); - if (endptr) - *(endptr + 2) = '\0'; - /* last object_id we parsed */ - sscanf(newmax, "%d", &end); + if (once) { + char *t = first_object_id_val(mbuf); + if (t && atoi(t) > start) + start = atoi(t); + free(t); + once = false; + } + /* clean up mbuf */ + endptr = strstr(mbuf, "{{{"); + if (endptr) + *(endptr + 2) = '\0'; + /* last object_id we parsed */ + sscanf(newmax, "%d", &end); #if UEMIS_DEBUG & 4 - fprintf(debugfile, "d_u_i after download and parse start %d end %d newmax %s progress %4.2f\n", start, end, newmax, progress_bar_fraction); + fprintf(debugfile, "d_u_i after download and parse start %d end %d newmax %s progress %4.2f\n", start, end, newmax, progress_bar_fraction); +#endif + /* The way this works is that I am reading the current dive from what has been loaded during the getDiveLogs call to the UEMIS. + * As the object_id of the divelog entry and the object_id of the dive details are not necessarily the same, the match needs + * to happen based on the logfilenr. + * What the following part does is to optimize the mapping by using + * dive_to_read = the dive deatils entry that need to be read using the object_id + * logFileNoToFind = map the logfilenr of the dive details with the object_id = diveid from the get dive logs */ + int dive_to_read = (lastFoundLogfilenr > 0 ? lastFoundLogfilenr + 1 : start); + td = data->download_table; + + for (int i = iMatchDiveAndLog; i < td->nr; i++) { + dive = td->dives[i]; + dTime = get_dive_date_c_string(dive->when); + snprintf(logFileNoToFind, sizeof(logFileNoToFind), "logfilenr{int{%d", dive->dc.diveid); + + bool found = false; + while (!found) { + snprintf(divetoRead, sizeof(divetoRead), "%d", dive_to_read); + param_buff[2] = divetoRead; + success = uemis_get_answer(mountpath, "getDive", 3, 0, &result); +#if UEMIS_DEBUG % 2 + do_dump_buffer_to_file(mbuf, strdup("Dive"), round); #endif - /* now download the additional dive data with "getDive" for the dives - * we just downloaded - yes, this is completely insane - why isn't all of - * this downloaded in the first place??? */ - for (i = start; i <= end; i++) { - snprintf(objectid, sizeof(objectid), "%d", i + offset); - param_buff[2] = objectid; + uemis_mem_status = get_memory(data->download_table); + if (uemis_mem_status == UEMIS_MEM_OK || uemis_mem_status == UEMIS_MEM_CRITICAL) { + /* if the memory isn's completely full we can try to read more divelog vs. dive details + * UEMIS_MEM_CRITICAL means not enough space for a full round but the dive details + * and the divespots should fit into the UEMIS memory + * The match we do here is to map the object_id to the logfilenr, we do this + * by iterating through the last set of loaded divelogs and then find the corresponding + * dive with the matching logfilenr */ + if (mbuf) { + if (strstr(mbuf, logFileNoToFind)) { + /* we found the logfilenr that matches our object_id from the divelog we were looking for + * we mark the search sucessfull even if the dive has been deleted. */ + found = true; + process_raw_buffer(data, deviceidnr, mbuf, &newmax, false, NULL); + if (strstr(mbuf, strdup("deleted{bool{true")) == NULL) { + /* remember the last log file number as it is very likely that subsequent dives + * have the same or higher logfile number. + * UEMIS unfortunately deletes dives by deleting the dive details and not the logs. */ #if UEMIS_DEBUG & 2 - fprintf(debugfile, "getDive %d, object_id %s\n", i, objectid); + fprintf(debugfile, "Matching divelog id %d from %s with dive details %d\n", dive->dc.diveid, dTime, iDiveToRead); #endif - /* there is no way I have found to directly get the dive information - * for dive #i as the object_id and logfilenr can be different in the - * getDive call; so we get the first one, compare the actual divenr - * with the one that we wanted, calculate the offset and try again. - * What an insane design... */ - success = uemis_get_answer(mountpath, "getDive", 3, 0, &result); - if (mbuf) { - int divenr; - (void)process_raw_buffer(data, deviceidnr, mbuf, &newmax, false, &divenr); + lastFoundLogfilenr = dive_to_read; + } else { + /* in this case we found a deleted file, so let's increment */ #if UEMIS_DEBUG & 2 - fprintf(debugfile, "got dive %d, looking for dive %d\n", divenr, i); + fprintf(debugfile, "TRY matching divelog id %d from %s with dive details %d but details are deleted\n", dive->dc.diveid, dTime, iDiveToRead); #endif - if (divenr != i) { - if (divenr == -1) { - offset--; - } else { - offset += i - divenr; - } + deletedFiles++; + /* mark this log entry as deleted and cleanup later, otherwise we mess up our array */ + dive->downloaded = false; #if UEMIS_DEBUG & 2 - fprintf(debugfile, " -> trying again with offset %d\n", offset); + fprintf(debugfile, "Deleted dive from %s, with id %d from table\n", dTime, dive->dc.diveid); #endif - i = start - 1; - if (i + offset < 0) + } + } else { + /* Ugly, need something better than this + * essentially, if we start reading divelogs not from the start + * we have no idea on how many log entries are there that have no + * valid dive details */ + if (dive_to_read >= dive->dc.diveid) + dive_to_read = (dive_to_read - 2 >= 0 ? dive_to_read - 2 : 0); + } + } + dive_to_read++; + } else { + /* At this point the memory of the UEMIS is full, let's cleanup all divelog files were + * we could not match the details to. */ + do_delete_dives(td, i); break; - continue; + } } + /* decrement iDiveToRead by the amount of deleted entries found to assure + * we are not missing any valid matches when processing subsequent logs */ + dive_to_read = (dive_to_read - deletedFiles > 0 ? dive_to_read - deletedFiles : 0); + deletedFiles = 0; + if (uemis_mem_status == UEMIS_MEM_FULL) + /* game over, not enough memory left */ + break; } - if (!success || import_thread_cancelled) + + /* + for (int i = iStartCleanup; i < data->download_table->nr; i++) + if (!data->download_table->dives[i]->downloaded) { + uemis_delete_dive(data, data->download_table->dives[i]->dc.diveid); + i = (i > iStartCleanup ? i-- : i = iStartCleanup); + } + */ + start = end; + + /* Do some memory checking here */ + uemis_mem_status = get_memory(data->download_table); + if (uemis_mem_status != UEMIS_MEM_OK) break; - } - start = end + 1; - /* if the user clicked cancel, exit gracefully */ - if (import_thread_cancelled) - goto bail; - /* if we got an error or got nothing back, stop trying */ - if (!success || !param_buff[3]) - break; - /* finally, if the memory is getting too full, maybe we better stop, too */ - if (progress_bar_fraction > 0.80) { - result = translate("gettextFromC", ERR_FS_ALMOST_FULL); + + /* if the user clicked cancel, exit gracefully */ + if (import_thread_cancelled) + break; + + /* if we got an error or got nothing back, stop trying */ + if (!success || !param_buff[3]) + break; + + /* finally, if the memory is getting too full, maybe we better stop, too */ + if (progress_bar_fraction > 0.80) { + result = translate("gettextFromC", ERR_FS_ALMOST_FULL); + break; + } + + +#if UEMIS_DEBUG & 2 + if (rounds != -1) + if (rounds-- == 0) + goto bail; +#endif + } else { + /* some of the loading from the UEMIS failed at the divelog level + * if the memory status = full, we cant even load the divespots and/or buddys. + * The loaded block of divelogs is useless and all new loaded divelogs need to + * be deleted from the download_table. + */ + if (uemis_mem_status == UEMIS_MEM_FULL) + do_delete_dives(data->download_table, iMatchDiveAndLog); break; } } + if (end == -2 && sscanf(newmax, "%d", &end) != 1) end = start; + #if UEMIS_DEBUG & 2 - fprintf(debugfile, "done: read from object_id %d to %d\n", first, end); + fprintf(debugfile, "Done: read from object_id %d to %d\n", first, end); #endif - free(newmax); - success = true; - for (i = 0; i <= nr_divespots; i++) { - char divespotnr[10]; - snprintf(divespotnr, sizeof(divespotnr), "%d", i); - param_buff[2] = divespotnr; + + /* Regardless on where we are with the memory situation, it's time now + * to see if we have to clean some dead bodies from our download table */ + next_table_index = 0; + while (data->download_table->dives[next_table_index]) { + if (!data->download_table->dives[next_table_index]->downloaded) + uemis_delete_dive(data, data->download_table->dives[next_table_index]->dc.diveid); + else + next_table_index++; + } + + switch (uemis_mem_status) { + case UEMIS_MEM_CRITICAL: + case UEMIS_MEM_OK: + for (i = 0; i <= nr_divespots; i++) { + char divespotnr[10]; + snprintf(divespotnr, sizeof(divespotnr), "%d", i); + param_buff[2] = divespotnr; +#if UEMIS_DEBUG & 2 + fprintf(debugfile, "getDivespot %d of %d, started at %d\n", i, nr_divespots, 0); +#endif + success = uemis_get_answer(mountpath, "getDivespot", 3, 0, &result); + if (mbuf && success) { #if UEMIS_DEBUG & 2 - fprintf(debugfile, "getDivespot %d\n", i); + do_dump_buffer_to_file(mbuf, strdup("Spot"), round); #endif - success = uemis_get_answer(mountpath, "getDivespot", 3, 0, &result); - if (mbuf) - parse_divespot(mbuf); + parse_divespot(mbuf); + } + } + if (uemis_mem_status == UEMIS_MEM_CRITICAL) + result = translate("gettextFromC", ERR_FS_ALMOST_FULL); + break; + case UEMIS_MEM_FULL: + result = translate("gettextFromC", ERR_FS_FULL); + break; } + bail: (void)uemis_get_answer(mountpath, "terminateSync", 0, 3, &result); if (!strcmp(param_buff[0], "error")) { -- 2.5.1
_______________________________________________ subsurface mailing list [email protected] http://lists.subsurface-divelog.org/cgi-bin/mailman/listinfo/subsurface
