Hello from DeviceAtlas
We are making available a patch to provide a device detection adaptor for HAProxy using DeviceAtlas. It will permit HAProxy users to know the nature of the visiting device, thus enabling different management according to the type of traffic. The output is an additional header containing information about the device (e.g. h/w type, OS, OS version, browser etc). Brief documentation is provided at doc/deviceatlas.txt, which links to https://deviceatlas.com/deviceatlas-haproxy-module, explaining how to retrieve the C API source and associated data file, how to compile the code and then how to configure HAProxy. Please cc me ( [email protected] ) on any responses. Best regards Tomas Trnka DeviceAtlas R&D Lead Afilias Technologies -- The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. dotMobi (Afilias Technologies Limited) is a private company limited by shares, incorporated and registered in the Republic of Ireland with registered number 398040 and registered office at First Floor, Fitzwilton House, Wilton Place, Dublin 2.
From: David Carlier <[email protected]> This diff is for the DeviceAtlas convertor. First, the configuration keywords handling (only the log level configuration part does not end the haproxy process if it is wrongly set, it fallbacks to the default level). Furthermore, init, deinit phases and the API lookup phase, the da_haproxy function which is fed by the input provided and set all necessary properties chosen via the configuration to the output, separated by the separator. diff --git a/include/import/da.h b/include/import/da.h new file mode 100644 index 0000000..d990188 --- /dev/null +++ b/include/import/da.h @@ -0,0 +1,13 @@ +#ifndef MOBI_DA_DAC_HAPROXY_H +#define MOBI_DA_DAC_HAPROXY_H +#ifdef USE_DEVICEATLAS + +#include <types/global.h> +#include <dac.h> + +void da_register_cfgkeywords(void); +int init_deviceatlas(void); +void deinit_deviceatlas(void); +int da_haproxy(const struct arg *, struct sample *, void *); +#endif +#endif diff --git a/src/da.c b/src/da.c new file mode 100644 index 0000000..148fefd --- /dev/null +++ b/src/da.c @@ -0,0 +1,231 @@ +#ifdef USE_DEVICEATLAS + +#include <stdio.h> + +#include <common/cfgparse.h> +#include <proto/log.h> +#include <import/da.h> + +static int da_json_file(char **args, int section_type, struct proxy *curpx, + struct proxy *defpx, const char *file, int line, + char **err) +{ + if (*(args[1]) == 0) { + memprintf(err, "deviceatlas json file : expects a json path.\n"); + return -1; + } + global.deviceatlas.jsonpath = strdup(args[1]); + return 0; +} + +static int da_log_level(char **args, int section_type, struct proxy *curpx, + struct proxy *defpx, const char *file, int line, + char **err) +{ + int loglevel; + if (*(args[1]) == 0) { + memprintf(err, "deviceatlas log level : expects an integer argument.\n"); + return -1; + } + + loglevel = atol(args[1]); + if (loglevel < 0 || loglevel > 3) { + memprintf(err, "deviceatlas log level : expects a log level between 0 and 3, %s given.\n", args[1]); + } else { + global.deviceatlas.loglevel = (da_severity_t)loglevel; + } + + return 0; +} + +static int da_property_separator(char **args, int section_type, struct proxy *curpx, + struct proxy *defpx, const char *file, int line, + char **err) +{ + if (*(args[1]) == 0) { + memprintf(err, "deviceatlas property separator : expects a character argument.\n"); + return -1; + } + global.deviceatlas.separator = *args[1]; + return 0; +} + +static struct cfg_kw_list dacfg_kws = {{ }, { + { CFG_GLOBAL, "deviceatlas-json-file", da_json_file }, + { CFG_GLOBAL, "deviceatlas-log-level", da_log_level }, + { CFG_GLOBAL, "deviceatlas-property-separator", da_property_separator }, + { 0, NULL, NULL }, +}}; + +static size_t da_haproxy_read(void *ctx, size_t len, char *buf) +{ + return fread(buf, 1, len, ctx); +} + +static da_status_t da_haproxy_seek(void *ctx, off_t off) +{ + return fseek(ctx, off, SEEK_SET) != -1 ? DA_OK : DA_SYS; +} + +static void da_haproxy_log(da_severity_t severity, da_status_t status, + const char *fmt, va_list args) +{ + if (severity <= global.deviceatlas.loglevel) { + char logbuf[256]; + vsnprintf(logbuf, sizeof(logbuf), fmt, args); + Warning("deviceatlas : %s.\n", logbuf); + } +} + +void da_register_cfgkeywords(void) +{ + cfg_register_keywords(&dacfg_kws); +} + +int init_deviceatlas(void) +{ + da_status_t status = DA_SYS; + if (global.deviceatlas.jsonpath != 0) { + FILE *jsonp; + da_property_decl_t extraprops[] = {{0, 0}}; + size_t atlasimglen; + da_status_t status; + + jsonp = fopen(global.deviceatlas.jsonpath, "r"); + if (jsonp == 0) { + Alert("deviceatlas : '%s' json file has invalid path or is not readable.\n", + global.deviceatlas.jsonpath); + goto out; + } + + da_init(); + da_seterrorfunc(da_haproxy_log); + status = da_atlas_compile(jsonp, da_haproxy_read, da_haproxy_seek, + &global.deviceatlas.atlasimgptr, &atlasimglen); + fclose(jsonp); + if (status != DA_OK) { + Alert("deviceatlas : '%s' json file is invalid.\n", + global.deviceatlas.jsonpath); + goto out; + } + + status = da_atlas_open(&global.deviceatlas.atlas, extraprops, + global.deviceatlas.atlasimgptr, atlasimglen); + + if (status != DA_OK) { + Alert("deviceatlas : data could not be compiled.\n"); + goto out; + } + + global.deviceatlas.useragentid = da_atlas_header_evidence_id(&global.deviceatlas.atlas, + "user-agent"); + + fprintf(stdout, "Deviceatlas module loaded.\n"); + } + +out: + return status == DA_OK; +} + +void deinit_deviceatlas(void) +{ + if (global.deviceatlas.jsonpath != 0) { + free(global.deviceatlas.jsonpath); + } + + if (global.deviceatlas.useragentid > 0) { + da_atlas_close(&global.deviceatlas.atlas); + free(global.deviceatlas.atlasimgptr); + } + + da_fini(); +} + +int da_haproxy(const struct arg *args, struct sample *smp, void *private) +{ + struct chunk *tmp; + da_deviceinfo_t devinfo; + da_propid_t prop, *pprop; + da_type_t proptype; + da_status_t status; + const char *useragent, *propname; + char useragentbuf[1024]; + int i; + + if (global.deviceatlas.useragentid == 0) { + return 1; + } + + tmp = get_trash_chunk(); + chunk_reset(tmp); + + memcpy(useragentbuf, smp->data.str.str, sizeof(useragentbuf)); + useragentbuf[smp->data.str.len] = 0; + + useragent = (const char *)useragentbuf; + propname = (const char *)args[0].data.str.str; + i = 0; + + status = da_search(&global.deviceatlas.atlas, &devinfo, + global.deviceatlas.useragentid, useragent, 0); + if (status != DA_OK) { + return 0; + } + + for (; propname != 0; i ++, propname = (const char *)args[i].data.str.str) { + status = da_atlas_getpropid(&global.deviceatlas.atlas, + propname, &prop); + if (status != DA_OK) { + chunk_appendf(tmp, "%c", global.deviceatlas.separator); + continue; + } + pprop = ∝ + da_atlas_getproptype(&global.deviceatlas.atlas, *pprop, &proptype); + + switch (proptype) { + case DA_TYPE_BOOLEAN: { + bool val; + status = da_getpropboolean(&devinfo, *pprop, &val); + if (status == DA_OK) { + chunk_appendf(tmp, "%d", val); + } + break; + } + case DA_TYPE_INTEGER: + case DA_TYPE_NUMBER: { + long val; + status = da_getpropinteger(&devinfo, *pprop, &val); + if (status == DA_OK) { + chunk_appendf(tmp, "%ld", val); + } + break; + } + case DA_TYPE_STRING: { + const char *val; + status = da_getpropstring(&devinfo, *pprop, &val); + if (status == DA_OK) { + chunk_appendf(tmp, "%s", val); + } + break; + } + default: + break; + } + + chunk_appendf(tmp, "%c", global.deviceatlas.separator); + } + + da_close(&devinfo); + + if (tmp->len) { + --tmp->len; + tmp->str[tmp->len] = 0; + } + + smp->data.str.str = tmp->str; + smp->data.str.len = strlen(tmp->str); + + return 1; +} + +#endif
From: David Carlier <[email protected]> This diff is the raw C struct definition of all DeviceAtlas module data needed added to the main global struct haproxy configuration. The three first members are needed for both init and deinit phases as some dynamic memory allocations are done. The useragentid serves to hold during the whole lifecycle of the module the User-Agent HTTP Header identifier from the DeviceAtlas data during the init process. diff --git a/include/types/global.h b/include/types/global.h index 7533bb0..8823128 100644 --- a/include/types/global.h +++ b/include/types/global.h @@ -31,6 +31,10 @@ #include <types/proxy.h> #include <types/task.h> +#ifdef USE_DEVICEATLAS +#include <dac.h> +#endif + #ifndef UNIX_MAX_PATH #define UNIX_MAX_PATH 108 #endif @@ -169,6 +173,16 @@ struct global { unsigned long cpu_map[LONGBITS]; /* list of CPU masks for the 32/64 first processes */ #endif struct proxy *stats_fe; /* the frontend holding the stats settings */ +#ifdef USE_DEVICEATLAS + struct { + void *atlasimgptr; + char *jsonpath; + da_atlas_t atlas; + da_evidence_id_t useragentid; + da_severity_t loglevel; + char separator; + } deviceatlas; +#endif }; extern struct global global;
From: David Carlier <[email protected]> This diff declares the deviceatlas module and can accept up to 5 property names for the API lookup. diff --git a/src/sample.c b/src/sample.c index 03540e1..37c5757 100644 --- a/src/sample.c +++ b/src/sample.c @@ -32,6 +32,10 @@ #include <proto/sample.h> #include <proto/stick_table.h> +#ifdef USE_DEVICEATLAS +#include <import/da.h> +#endif + /* sample type names */ const char *smp_to_type[SMP_TYPES] = { [SMP_T_BOOL] = "bool", @@ -2278,6 +2282,9 @@ static struct sample_conv_kw_list sample_conv_kws = {ILH, { { "div", sample_conv_arith_div, ARG1(1,UINT), NULL, SMP_T_UINT, SMP_T_UINT }, { "mod", sample_conv_arith_mod, ARG1(1,UINT), NULL, SMP_T_UINT, SMP_T_UINT }, { "neg", sample_conv_arith_neg, 0, NULL, SMP_T_UINT, SMP_T_UINT }, +#ifdef USE_DEVICEATLAS + { "da-csv", da_haproxy, ARG5(1,STR,STR,STR,STR,STR), NULL, SMP_T_STR, SMP_T_STR }, +#endif { NULL, NULL, 0, 0, 0 }, }};
From: David Carlier <[email protected]> This diff initialises few DeviceAtlas struct fields member with their inherent default values. Furthermore, the specific DeviceAtlas configuration keywords are registered and the module is initialised and all necessary resources are freed during the deinit phase. diff --git a/src/haproxy.c b/src/haproxy.c index 233c434..2cdcdeb 100644 --- a/src/haproxy.c +++ b/src/haproxy.c @@ -111,6 +111,10 @@ #include <proto/ssl_sock.h> #endif +#ifdef USE_DEVICEATLAS +#include <import/da.h> +#endif + /*********************************************************************/ extern const struct comp_algo comp_algos[]; @@ -170,6 +174,14 @@ struct global global = { .maxsslconn = DEFAULT_MAXSSLCONN, #endif #endif +#ifdef USE_DEVICEATLAS + .deviceatlas = { + .loglevel = DA_SEV_INFO, + .useragentid = 0, + .jsonpath = 0, + .separator = '|', + }, +#endif /* others NULL OK */ }; @@ -571,6 +583,10 @@ void init(int argc, char **argv) /* Initialise lua. */ hlua_init(); +#if defined(USE_DEVICEATLAS) + /* Register deviceatlas config keywords */ + da_register_cfgkeywords(); +#endif global.tune.options |= GTUNE_USE_SELECT; /* select() is always available */ #if defined(ENABLE_POLL) @@ -782,6 +798,9 @@ void init(int argc, char **argv) /* now we know the buffer size, we can initialize the channels and buffers */ init_buffer(); +#if defined(USE_DEVICEATLAS) + init_deviceatlas(); +#endif if (have_appsession) appsession_init(); @@ -1414,6 +1433,10 @@ void deinit(void) protocol_unbind_all(); +#if defined(USE_DEVICEATLAS) + deinit_deviceatlas(); +#endif + free(global.log_send_hostname); global.log_send_hostname = NULL; free(global.log_tag); global.log_tag = NULL; free(global.chroot); global.chroot = NULL;
From: David Carlier <[email protected]> This diff updates the Makefile to compile conditionally via some new sets of flags, USE_DEVICEATLAS to enable the module and the couple DEVICEATLAS_INC/DEVICEATLAS_LIB which needs to point to the API root folder in order to compile the API and the module. diff --git a/Makefile b/Makefile index c0f4cb3..5d583be 100644 --- a/Makefile +++ b/Makefile @@ -38,6 +38,7 @@ # USE_TFO : enable TCP fast open. Supported on Linux >= 3.7. # USE_NS : enable network namespace support. Supported on Linux >= 2.6.24. # USE_DL : enable it if your system requires -ldl. Automatic on Linux. +# USE_DEVICEATLAS : enable DeviceAtlas api. # # Options can be forced by specifying "USE_xxx=1" or can be disabled by using # "USE_xxx=" (empty string). @@ -666,6 +667,17 @@ OPTIONS_CFLAGS += -DCONFIG_HAP_NS BUILD_OPTIONS += $(call ignore_implicit,USE_NS) endif + +ifneq ($(USE_DEVICEATLAS),) +DEVICEATLAS_INC = +DEVICEATLAS_LIB = +OPTIONS_OBJS += $(DEVICEATLAS_LIB)/json.o +OPTIONS_OBJS += $(DEVICEATLAS_LIB)/dac.o +OPTIONS_OBJS += src/da.o +OPTIONS_CFLAGS += -DUSE_DEVICEATLAS $(if $(DEVICEATLAS_INC),-I$(DEVICEATLAS_INC)) +BUILD_OPTIONS += $(call ignore_implicit,USE_DEVICEATLAS) +endif + #### Global link options # These options are added at the end of the "ld" command line. Use LDFLAGS to # add options at the beginning of the "ld" command line if needed.
From: David Carlier <[email protected]> This diff is related to the additional documentation in order to build the DeviceAtlas module and in addition with an example of a basic configuration. diff --git a/doc/deviceatlas.txt b/doc/deviceatlas.txt new file mode 100644 index 0000000..cb6c618 --- /dev/null +++ b/doc/deviceatlas.txt @@ -0,0 +1,22 @@ +In order to add DeviceAtlas Device Detection support, you would need to download +the API source code from https://deviceatlas.com/deviceatlas-haproxy-module and once extracted : + +$ make TARGET=<target> USE_PCRE=1 USE_DEVICEATLAS=1 DEVICEATLAS_INC=<path to the API root folder> DEVICEATLAS_LIB=<path to the API root folder> + +These are supported DeviceAtlas directives : + +- deviceatlas-json-file <path to the DeviceAtlas JSON data file>. +- deviceatlas-log-level <number> (0 to 3, level of information returned by the API, 0 by default). +- deviceatlas-property-separator <character> (character used to separate the properties produced by the API, | by default). + +Sample configuration : + +global + deviceatlas-json-file <path to json file> + +... +frontend + bind *:8881 + default_backend servers + http-request set-header X-DeviceAtlas-Data %[req.fhdr(User-Agent),da-csv(primaryHardwareType,osName,osVersion,browserName,browserVersion)] +
From: David Carlier <[email protected]> This diff updates the main configuration readme file with the specific DeviceAtlas module keywords. diff --git a/doc/configuration.txt b/doc/configuration.txt index 7533bb0..8823128 100644 --- a/doc/configuration.txt 2015-05-29 17:37:38.350265994 +0100 +++ b/doc/configuration.txt 2015-05-29 17:17:50.970293592 +0100 @@ -10646,6 +10646,18 @@ not be used for security purposes as a 32-bit hash is trivial to break. See also "djb2", "sdbm", "wt6" and the "hash-type" directive. +deviceatlas-json-file(<path>) + Sets the path of the DeviceAtlas JSON data file to be loaded by the API. + The path must be a valid JSON data file and accessible by Haproxy process. + +deviceatlas-log-level(<value>) + Sets the level of informations returned by the API. This directive is + optional and set to 0 by default if not set. + +deviceatlas-separator(<char>) + Sets the character separator for the API properties results. This directive + is optional and set to | by default if not set. + div(<value>) Divides the input value of type unsigned integer by <value>, and returns the result as an unsigned integer. If <value> is null, the largest unsigned
