This is an automated email from the ASF dual-hosted git repository. sergehuber pushed a commit to branch UNOMI-1023-UNOMI-1024-compat-fixes in repository https://gitbox.apache.org/repos/asf/unomi-tracker.git
commit 6a86de038ae4fe3a6000e71afe596531213860e6 Author: Serge Huber <[email protected]> AuthorDate: Mon Oct 26 17:42:15 2026 +0100 UNOMI-1023, UNOMI-1024: Check compatibility with 1.5.0 and Unomi 2.3 in the server tests - compare-1.5.0: loads the published 1.5.0 and the current tracker on the same page and compares their requests; they must be identical, except for the empty page "attributes" no longer sent - unomi2-legacy: Unomi 2.3.0, the last version that accepts an empty page "attributes" - The test page no longer adds a Buffer shim (only for 1.5.0, whose users always have a polyfill), and the two known-failure tests are now normal tests Co-Authored-By: Claude Opus 5.5 <[email protected]> --- .gitignore | 1 + test/servers/README.md | 35 ++++++----- test/servers/compare-1.5.0.spec.js | 97 +++++++++++++++++++++++++++++++ test/servers/docker-compose.yml | 14 ++++- test/servers/page/index.html | 20 ++++--- test/servers/playwright.compare.config.js | 28 +++++++++ test/servers/run.js | 31 ++++++++-- test/servers/serve.js | 4 +- test/servers/tracker.spec.js | 27 +++++---- test/servers/unomi.js | 2 + 10 files changed, 219 insertions(+), 40 deletions(-) diff --git a/.gitignore b/.gitignore index cf71b0f..d11e8e9 100644 --- a/.gitignore +++ b/.gitignore @@ -275,3 +275,4 @@ yalc.lock # Tracker server tests (test/servers) test/servers/test-results/ playwright-report/ +test/servers/.cache/ diff --git a/test/servers/README.md b/test/servers/README.md index dcf4ef5..23d3f9d 100644 --- a/test/servers/README.md +++ b/test/servers/README.md @@ -33,25 +33,28 @@ Requirements Running ------- ```shell -yarn test:servers # every server, one after the other +yarn test:servers # the comparison with 1.5.0, then every server yarn test:servers unomi2 unomi4 # only these servers +yarn test:servers compare-1.5.0 # only the comparison with 1.5.0 (no Docker needed) yarn test:servers --keep unomi4 # leave the server running afterwards ``` -The command builds the tracker, then for each server: starts it, sets it up, runs the tests and -removes it with its data. It exits with an error if any server fails. +The command builds the tracker and compares its requests with those of the published 1.5.0. Then, +for each server, it starts the server, sets it up, runs the tests and removes the server with its +data. It exits with an error if anything fails. Servers ------- | Name | Server | Search engine | Setup before the tests | |---|---|---|---| +| `unomi2-legacy` | `apache/unomi:2.3.0`, the last version that accepts an empty page `attributes` | Elasticsearch 7.17 | Scope `tracker-test` | | `unomi2` | `apache/unomi:2.7.0` | Elasticsearch 7.17 | Scope `tracker-test` | | `unomi3` | `apache/unomi:3.0.0` | Elasticsearch 9.1 | Scope `tracker-test` | | `unomi4` | `apache/unomi:4.0.0-SNAPSHOT`, multi-tenant | Elasticsearch 9.4 | Tenant `tracker-test`, new public and private keys, scope `tracker-test` in the tenant | | `unomi4-compat` | `apache/unomi:4.0.0-SNAPSHOT`, single-tenant compatibility mode | Elasticsearch 9.4 | Tenant `default`, scope `tracker-test` in it | -Unomi listens on http://localhost:8181. Override the images with `UNOMI2_VERSION`, `UNOMI3_VERSION` -and `UNOMI4_VERSION`, for example `UNOMI4_VERSION=4.0.1-SNAPSHOT yarn test:servers unomi4`. +Unomi listens on http://localhost:8181. Override the images with `UNOMI2_LEGACY_VERSION`, +`UNOMI2_VERSION`, `UNOMI3_VERSION` and `UNOMI4_VERSION`, for example `UNOMI4_VERSION=4.0.1-SNAPSHOT yarn test:servers unomi4`. The Unomi 4 servers use the administrator password `tracker-test-root` (`UNOMI_ROOT_PASSWORD`). With `--keep`, the setup log shows the tenant public key, so you can try the server by hand. @@ -61,9 +64,15 @@ cookies (`UNOMI_PUBLIC_CORS_ALLOWED_ORIGINS`) and does not list http://localhost What is tested -------------- +Compatibility with existing sites (`compare-1.5.0`): the current tracker and the published 1.5.0 are +loaded on the same page with the same configuration, and their requests are compared (URLs, headers +and bodies). They must be identical, except that an empty page `attributes` is no longer sent. +1.5.0 gets a `Buffer` polyfill, as sites using it always have one. + For each server: - The context loads and a click event is stored, with the configuration a site uses for that server (`apiKey` on Unomi 4 multi-tenant only), and in the expected tenant on Unomi 4. +- The browser build runs without a `Buffer` polyfill. - Setting `apiKey` works on every server, and the `X-Unomi-Api-Key` header is only sent when set. - Unomi 4 multi-tenant answers `401` without a key. - With cookies, the profile is kept across page loads. @@ -71,20 +80,16 @@ For each server: Unomi 2.x and 3.x keep the profile through the session. - From an origin the server does not list, Unomi 4 blocks requests with cookies, and accepts them without cookies. - -Two tests are marked as known failures (`test.fail()`), for bugs that are not fixed yet. Playwright -reports them as passed while the bug is there, and fails them once it is fixed, so that the marker -gets removed: -- The UMD build throws `Buffer is not defined` in a browser: es6-crawler-detect calls - `Buffer.byteLength`. The test page adds a small shim so that the other tests can run. -- Events from a page whose `digitalData.page` has `attributes` (as the Unomi manual tutorial - showed) are dropped by the server: the page schema does not allow that property since - UNOMI-790 (Unomi 2.4.0). +- A page with empty `attributes` (as the Unomi manual tutorial showed) gets its events stored. + Non-empty `attributes` are sent as before; no default page schema accepts them, so the server + drops the event, as with 1.5.0. Unomi 2.0 to 2.3 accept an empty one; 2.4 and later reject it + too (UNOMI-790). Files ----- - `docker-compose.yml`: the servers, one Compose profile each - `run.js`: starts each server and runs the tests (`yarn test:servers`) - `unomi.js`: server setup (tenant, keys, scope) and event queries -- `tracker.spec.js`: the tests +- `tracker.spec.js`: the tests against each server +- `compare-1.5.0.spec.js`: the comparison with the published 1.5.0 - `page/index.html`, `serve.js`: the test page, configured through its query string diff --git a/test/servers/compare-1.5.0.spec.js b/test/servers/compare-1.5.0.spec.js new file mode 100644 index 0000000..1f49cdc --- /dev/null +++ b/test/servers/compare-1.5.0.spec.js @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Compares the requests of the current tracker with those of the published 1.5.0, for the same +// page and configuration, so that existing sites keep sending the same data. The test answers +// the requests itself: no Unomi server is needed. Run through `yarn test:servers`. +const { test, expect } = require('@playwright/test'); + +const UNOMI_URL = 'http://localhost:8181'; +const ORIGIN = 'http://localhost:8000'; + +async function captureRequests(page, tracker, config) { + const requests = []; + await page.route(UNOMI_URL + '/**', async route => { + const request = route.request(); + const headers = await request.allHeaders(); + requests.push({ + method: request.method(), + url: request.url(), + contentType: headers['content-type'], + accept: headers['accept'], + apiKey: headers['x-unomi-api-key'], + body: request.postData() ? JSON.parse(request.postData()) : undefined + }); + await route.fulfill({ + status: 200, + headers: { 'Access-Control-Allow-Origin': ORIGIN, 'Access-Control-Allow-Credentials': 'true', 'Content-Type': 'application/json' }, + body: JSON.stringify({ profileId: 'compare-profile', sessionId: 'compare-session' }) + }); + }); + if (tracker) { + await page.context().addCookies([{ name: 'tracker', value: tracker, url: ORIGIN }]); + } + const params = new URLSearchParams(Object.assign({ unomi: UNOMI_URL, sessionId: 'compare-session' }, config)); + await page.goto(ORIGIN + '/?' + params); + const context = await page.evaluate(() => window.trackerTest.context); + expect(context.loaded).toBe(true); + const click = await page.evaluate(() => window.trackerTest.sendClick('compared-click')); + expect(click.sent).toBe(true); + return requests; +} + +// Removes an empty "attributes" from the page items, the only change the current tracker makes +function withoutEmptyPageAttributes(value) { + if (Array.isArray(value)) { + return value.map(withoutEmptyPageAttributes); + } + if (value && typeof value === 'object') { + const copy = {}; + for (const key of Object.keys(value)) { + copy[key] = withoutEmptyPageAttributes(value[key]); + } + if (copy.itemType === 'page' && copy.properties && copy.properties.attributes + && typeof copy.properties.attributes === 'object' && Object.keys(copy.properties.attributes).length === 0) { + delete copy.properties.attributes; + } + return copy; + } + return value; +} + +const cases = [ + { name: 'a page without attributes', config: {}, same: true }, + { name: 'a page with non-empty attributes', config: { pageAttributes: '{"color":"blue"}' }, same: true }, + { name: 'a page with empty attributes', config: { pageAttributes: '{}' }, same: false } +]; + +for (const { name, config, same } of cases) { + test('sends the same requests as 1.5.0 for ' + name, async ({ browser }) => { + const published = await browser.newPage(); + const current = await browser.newPage(); + const before = await captureRequests(published, '1.5.0', config); + const after = await captureRequests(current, undefined, config); + expect(before.map(request => request.url)).toEqual([UNOMI_URL + '/context.json', UNOMI_URL + '/eventcollector']); + expect(after).toEqual(same ? before : withoutEmptyPageAttributes(before)); + if (!same) { + // The difference must be real: 1.5.0 sends the empty attributes, the current tracker does not + expect(after).not.toEqual(before); + } + await published.close(); + await current.close(); + }); +} diff --git a/test/servers/docker-compose.yml b/test/servers/docker-compose.yml index 1fc23b4..b50a138 100644 --- a/test/servers/docker-compose.yml +++ b/test/servers/docker-compose.yml @@ -36,7 +36,7 @@ services: elasticsearch7: <<: *elasticsearch image: docker.elastic.co/elasticsearch/elasticsearch:${ES7_VERSION:-7.17.28} - profiles: ["unomi2"] + profiles: ["unomi2-legacy", "unomi2"] elasticsearch9-1: <<: *elasticsearch @@ -48,6 +48,18 @@ services: image: docker.elastic.co/elasticsearch/elasticsearch:${ES9_4_VERSION:-9.4.3} profiles: ["unomi4", "unomi4-compat"] + # Unomi 2.3 is the last version whose page schema allows an empty "attributes" (UNOMI-790) + unomi2-legacy: + image: apache/unomi:${UNOMI2_LEGACY_VERSION:-2.3.0} + profiles: ["unomi2-legacy"] + environment: + - UNOMI_ELASTICSEARCH_ADDRESSES=elasticsearch7:9200 + ports: + - "${UNOMI_PORT:-8181}:8181" + depends_on: + elasticsearch7: + condition: service_healthy + unomi2: image: apache/unomi:${UNOMI2_VERSION:-2.7.0} profiles: ["unomi2"] diff --git a/test/servers/page/index.html b/test/servers/page/index.html index 69f07c3..04f35fe 100644 --- a/test/servers/page/index.html +++ b/test/servers/page/index.html @@ -20,17 +20,20 @@ <meta charset="utf-8"> <title>Unomi tracker test page</title> <script> - // The crawler detection library calls Buffer.byteLength, which browsers do not have. - // ?noBufferShim loads the tracker without this shim, to check the bug is still there. - if (!new URLSearchParams(document.location.search).has('noBufferShim') && typeof window.Buffer === 'undefined') { + // A "tracker=1.5.0" cookie loads the published 1.5.0 tracker instead of the current build (a cookie, + // because the tracker sends the URL parameters). Sites using 1.5.0 always have a Buffer polyfill + // (it fails without one), so add one for it only. + if (document.cookie.split('; ').indexOf('tracker=1.5.0') !== -1) { window.Buffer = { byteLength: function (value) { return new TextEncoder().encode(value).length; } }; + document.write('<script src="/tracker-1.5.0.js"><\/script>'); + } else { + document.write('<script src="/tracker.js"><\/script>'); } </script> - <script src="/tracker.js"></script> </head> <body> <script> @@ -39,7 +42,8 @@ // apiKey wemInitConfig.apiKey (optional) // sendCredentials wemInitConfig.sendCredentials, "true" or "false" (optional) // noBufferShim load the tracker without the Buffer shim (optional) - // pageAttributes add "attributes": {} to digitalData.page, as the Unomi manual does (optional) + // pageAttributes digitalData.page.attributes as JSON, for example {} (optional) + // sessionId session ID to use instead of a new one (optional) (function () { const params = new URLSearchParams(document.location.search); const wemInitConfig = { @@ -76,11 +80,13 @@ }; if (params.has('pageAttributes')) { - digitalData.page.attributes = {}; + digitalData.page.attributes = JSON.parse(params.get('pageAttributes')); } const tracker = window['apache-unomi-tracker'].useTracker(); - if (tracker.getCookie(wemInitConfig.trackerSessionIdCookieName) == null) { + if (params.has('sessionId')) { + tracker.setCookie(wemInitConfig.trackerSessionIdCookieName, params.get('sessionId'), 1); + } else if (tracker.getCookie(wemInitConfig.trackerSessionIdCookieName) == null) { tracker.setCookie(wemInitConfig.trackerSessionIdCookieName, tracker.generateGuid(), 1); } diff --git a/test/servers/playwright.compare.config.js b/test/servers/playwright.compare.config.js new file mode 100644 index 0000000..f29a32f --- /dev/null +++ b/test/servers/playwright.compare.config.js @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Comparison with the published 1.5.0 tracker (compare-1.5.0.spec.js). No Unomi server needed. +// Run through `yarn test:servers`, which downloads 1.5.0 first. +const path = require('path'); +const { defineConfig } = require('@playwright/test'); +const base = require('./playwright.config'); + +module.exports = defineConfig(Object.assign({}, base, { + testMatch: 'compare-1.5.0.spec.js', + outputDir: path.join(__dirname, 'test-results', 'compare-1.5.0'), + globalSetup: undefined +})); diff --git a/test/servers/run.js b/test/servers/run.js index 9979cdd..8d41235 100644 --- a/test/servers/run.js +++ b/test/servers/run.js @@ -17,12 +17,14 @@ // Runs the browser tests against each Unomi server version, one after the other. // -// yarn test:servers all servers +// yarn test:servers the comparison with 1.5.0, then all servers // yarn test:servers unomi2 unomi4 only these servers +// yarn test:servers compare-1.5.0 only the comparison with 1.5.0 (no Docker needed) // yarn test:servers --keep unomi4 leave the server running afterwards // // Not part of CI: it needs Docker and takes several minutes. const { spawnSync } = require('child_process'); +const fs = require('fs'); const path = require('path'); const { TARGETS } = require('./unomi'); @@ -30,12 +32,13 @@ const root = path.join(__dirname, '..', '..'); const composeFile = path.join(__dirname, 'docker-compose.yml'); const args = process.argv.slice(2); const keep = args.includes('--keep'); +const COMPARE = 'compare-1.5.0'; const names = args.filter(arg => !arg.startsWith('--')); -const selected = names.length > 0 ? names : Object.keys(TARGETS); +const selected = names.length > 0 ? names : [COMPARE].concat(Object.keys(TARGETS)); for (const name of selected) { - if (!TARGETS[name]) { - console.error('Unknown server "' + name + '", expected one of: ' + Object.keys(TARGETS).join(', ')); + if (name !== COMPARE && !TARGETS[name]) { + console.error('Unknown name "' + name + '", expected one of: ' + [COMPARE].concat(Object.keys(TARGETS)).join(', ')); process.exit(2); } } @@ -54,8 +57,26 @@ if (!run('yarn', ['build'])) { process.exit(1); } +// The published 1.5.0 tracker, for the comparison +function downloadPublishedTracker() { + const directory = path.join(__dirname, '.cache', 'apache-unomi-tracker-1.5.0'); + if (fs.existsSync(path.join(directory, 'package', 'dist', 'apache-unomi-tracker.umd.js'))) { + return true; + } + fs.mkdirSync(directory, { recursive: true }); + return run('npm', ['pack', '[email protected]', '--pack-destination', directory]) + && run('tar', ['xzf', path.join(directory, 'apache-unomi-tracker-1.5.0.tgz'), '-C', directory]); +} + const results = []; -for (const name of selected) { +if (selected.includes(COMPARE)) { + console.log('\n=== ' + COMPARE); + const passed = downloadPublishedTracker() + && run('npx', ['playwright', 'test', '-c', path.join(__dirname, 'playwright.compare.config.js')]); + results.push({ name: COMPARE, passed }); +} + +for (const name of selected.filter(name => name !== COMPARE)) { console.log('\n=== ' + name); let passed = false; try { diff --git a/test/servers/serve.js b/test/servers/serve.js index c97f476..95f8591 100644 --- a/test/servers/serve.js +++ b/test/servers/serve.js @@ -25,7 +25,9 @@ const path = require('path'); const PORTS = [8000, 8001]; const files = { '/': { file: path.join(__dirname, 'page', 'index.html'), type: 'text/html; charset=utf-8' }, - '/tracker.js': { file: path.join(__dirname, '..', '..', 'dist', 'apache-unomi-tracker.umd.js'), type: 'text/javascript; charset=utf-8' } + '/tracker.js': { file: path.join(__dirname, '..', '..', 'dist', 'apache-unomi-tracker.umd.js'), type: 'text/javascript; charset=utf-8' }, + // Downloaded by run.js, for the comparison with the published 1.5.0 + '/tracker-1.5.0.js': { file: path.join(__dirname, '.cache', 'apache-unomi-tracker-1.5.0', 'package', 'dist', 'apache-unomi-tracker.umd.js'), type: 'text/javascript; charset=utf-8' } }; function handle(request, response) { diff --git a/test/servers/tracker.spec.js b/test/servers/tracker.spec.js index a57a55d..d340cd4 100644 --- a/test/servers/tracker.spec.js +++ b/test/servers/tracker.spec.js @@ -61,23 +61,28 @@ async function expectStoredClick(click) { } test.describe(server.name, () => { - test('the browser build runs without Node polyfills', async ({ page }) => { - // Known bug: es6-crawler-detect calls Buffer.byteLength, so the UMD build throws in a browser. - // Remove test.fail() once the build works without the shim. - test.fail(); + test('the browser build runs without a Buffer polyfill', async ({ page }) => { const errors = []; page.on('pageerror', error => errors.push(error.message)); - const { context } = await openPage(page, LISTED_ORIGIN, Object.assign({ noBufferShim: 'true' }, siteConfig), 10000); + const { context } = await openPage(page, LISTED_ORIGIN, siteConfig); expect(errors).toEqual([]); expect(context.loaded).toBe(true); }); - test('collects events from a page with "attributes", as the Unomi manual shows', async ({ page }) => { - // Known bug: the tracker sends digitalData.page as the page properties, and the page schema - // does not allow "attributes", so Unomi drops the events. Remove test.fail() once fixed. - test.fail(); - await openPage(page, LISTED_ORIGIN, Object.assign({ pageAttributes: 'true' }, siteConfig)); - await expectStoredClick(await sendClick(page, 'page-attributes')); + test('collects events from a page with empty "attributes", as the Unomi manual showed', async ({ page }) => { + const { context } = await openPage(page, LISTED_ORIGIN, Object.assign({ pageAttributes: '{}' }, siteConfig)); + expect(context.loaded).toBe(true); + await expectStoredClick(await sendClick(page, 'empty-page-attributes')); + }); + + test('sends non-empty page "attributes" as before', async ({ page }) => { + // No default page schema accepts attributes with content, so the server drops the event, as with + // 1.5.0 (compare-1.5.0 checks that the request is the same). Unomi 2.0 to 2.3 accept an empty one. + await openPage(page, LISTED_ORIGIN, Object.assign({ pageAttributes: '{"color":"blue"}' }, siteConfig)); + const click = await sendClick(page, 'page-attributes'); + expect(click.sent).toBe(true); + const events = await findClickEvents(process.env.UNOMI_READ_CREDENTIALS, click.targetId, 5000); + expect(events).toHaveLength(0); }); test('loads the context and collects an event with the site configuration', async ({ page }) => { diff --git a/test/servers/unomi.js b/test/servers/unomi.js index 7b63654..6a9d288 100644 --- a/test/servers/unomi.js +++ b/test/servers/unomi.js @@ -23,7 +23,9 @@ const SCOPE = 'tracker-test'; // Unomi 2.x and 3.x have no tenants and accept cookies from any origin. // Unomi 4 only accepts cookies from listed origins (here http://localhost:8000, not http://localhost:8001). +// Unomi 2.0 to 2.3 accept an empty page "attributes"; 2.4 removed it from the page schema (UNOMI-790). const TARGETS = { + 'unomi2-legacy': { major: 2, multiTenant: false, admin: 'karaf:karaf' }, 'unomi2': { major: 2, multiTenant: false, admin: 'karaf:karaf' }, 'unomi3': { major: 3, multiTenant: false, admin: 'karaf:karaf' }, 'unomi4': { major: 4, multiTenant: true, admin: 'karaf:' + ROOT_PASSWORD },
