Hi, We use Nginx to locally cache large binary content. Cache efficiency is of high importance, so the slice filter is excellent for this purpose! Here's a (very) short relevant configuration snippet for a particular location we use:
slice 1m; proxy_cache data; proxy_pass http://$host$request_uri; proxy_cache_key $uri$slice_range; proxy_set_header Range $slice_range; Since we've adopted the slice filter, we've been seeing a lot of 'etag mismatch in slice response' errors because the upstream provider serves content from multiple CDNs that provide the exact same URI namespace. For example, consider the following resource: /tpr/hs/data/7c/bf/7cbf4dbce9233a57ff6c1cbf84db77d8 Level3 serves this resource with ETag 74bc439-54261c855fd64. Akamai serves this resource with ETag 1273157742232ec1b8d3afc4c41a2a9c:1480362585. Since we value cache efficiency over anything else, adding the host to the cache key is not an option. The alternative is pinning proxy_pass to one specific CDN, but that approach has several other downsides. So, I decided to try my hand at (slightly) altering the filter to make it more configurable, I've posted the result below. Further improvements might include: - A (regex?) transformation for the ETag field, but that wouldn't be useful in the scenario above - exporting the ETag field out of the slice filter as a variable, effectively making the cache mutable 'slice_etag' might be slightly too obscure to people unfamiliar with the internals, perhaps 'slice_match_etag' would be more descriptive, but I'll leave that up to the maintainers. I've seen 'ngx_flag_t' in other *_conf_t typedefs, I hope that's appropriate. I'm not a C programmer, but I did the best I could. Any feedback is highly appreciated. Take care, Timo # HG changeset patch # User Timo Beckers <[email protected]> # Date 1482843958 -3600 # Tue Dec 27 14:05:58 2016 +0100 # Node ID 9e8c70191f0a9db9ad08bfb959fc1b1404d312ec # Parent 54cf51c4f07add000b824036d4dce4bc60e67bcf Slice filter: add slice_etag config var to disable etag handling When caching 206 requests from multiple CDNs that serve identical resources but have a different ETag scheme, this setting can be used to disable the ETag checking during processing of response headers. This comes in handy when seeing an excessive amount of 'etag mismatch in slice response' errors and the cache is short-lived enough to ensure clients don't receive stale content. diff -r 54cf51c4f07a -r 9e8c70191f0a src/http/modules/ngx_http_slice_filter_module.c --- a/src/http/modules/ngx_http_slice_filter_module.c Mon Dec 26 14:27:05 2016 +0300 +++ b/src/http/modules/ngx_http_slice_filter_module.c Tue Dec 27 14:05:58 2016 +0100 @@ -12,6 +12,7 @@ typedef struct { size_t size; + ngx_flag_t etag; } ngx_http_slice_loc_conf_t; @@ -48,6 +49,13 @@ static ngx_command_t ngx_http_slice_filter_commands[] = { + { ngx_string("slice_etag"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_slice_loc_conf_t, etag), + NULL }, + { ngx_string("slice"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, ngx_conf_set_size_slot, @@ -123,22 +131,26 @@ return NGX_ERROR; } - h = r->headers_out.etag; + slcf = ngx_http_get_module_loc_conf(r, ngx_http_slice_filter_module); - if (ctx->etag.len) { - if (h == NULL - || h->value.len != ctx->etag.len - || ngx_strncmp(h->value.data, ctx->etag.data, ctx->etag.len) - != 0) - { - ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, - "etag mismatch in slice response"); - return NGX_ERROR; + if (slcf->etag) { + h = r->headers_out.etag; + + if (ctx->etag.len) { + if (h == NULL + || h->value.len != ctx->etag.len + || ngx_strncmp(h->value.data, ctx->etag.data, ctx->etag.len) + != 0) + { + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "etag mismatch in slice response"); + return NGX_ERROR; + } } - } - if (h) { - ctx->etag = h->value; + if (h) { + ctx->etag = h->value; + } } if (ngx_http_slice_parse_content_range(r, &cr) != NGX_OK) { @@ -157,8 +169,6 @@ "http slice response range: %O-%O/%O", cr.start, cr.end, cr.complete_length); - slcf = ngx_http_get_module_loc_conf(r, ngx_http_slice_filter_module); - end = ngx_min(cr.start + (off_t) slcf->size, cr.complete_length); if (cr.start != ctx->start || cr.end != end) { @@ -480,6 +490,7 @@ } slcf->size = NGX_CONF_UNSET_SIZE; + slcf->etag = NGX_CONF_UNSET; return slcf; } @@ -492,6 +503,7 @@ ngx_http_slice_loc_conf_t *conf = child; ngx_conf_merge_size_value(conf->size, prev->size, 0); + ngx_conf_merge_off_value(conf->etag, prev->etag, 1); return NGX_CONF_OK; } _______________________________________________ nginx-devel mailing list [email protected] http://mailman.nginx.org/mailman/listinfo/nginx-devel
