CritasWang opened a new pull request, #51: URL: https://github.com/apache/tsfile-viewer/pull/51
Fixes #45 ## Problem `exportCSV()` / `exportJSON()` serialized `dataRows`, which only holds the current preview page, so the exported subset changed whenever the user paged. The user guide promises the export covers every record matching the filters. Paging the preview endpoint from the browser is not enough: with post-read filters the backend caps a single read at `query.max-result-size` (10,000), so the export still truncated there — measured 10,000 rows on a 16,031-row file. ## Changes - New `POST /api/data/export?format=csv|json` streams the whole filtered result set. The server pages through the file itself (2,000 rows per page, offset pushed down), applies exactly the same filters as the preview, and flushes after every page, so the file never has to fit in memory and `max-result-size` no longer truncates. `limit`/`offset` in the request body are ignored. - CSV follows RFC 4180 (commas, quotes, line breaks quoted and escaped), columns are the file's column union in lexicographic order, and timestamps are ISO-8601 UTC with milliseconds — the same text the browser used to produce. - JSON streams one row object per line, identical in shape to the preview rows. - Frontend requests the blob from the new endpoint and keeps the export buttons disabled while it runs. - `DataPreviewRequest` now accepts `advancedConditions`: the frontend has been sending it, but the backend silently dropped it, so advanced filters never applied to the preview either. ## Verification Against a 16,031-row tree-model file in a local instance, driven through the real UI with Playwright: | | before | after | |---|---|---| | rows on the preview page | 100 | 100 | | CSV data rows | 100 | **16,031** | | JSON rows | 100 | **16,031** | - ~100 ms for either format including the download; no console errors. - `limit: 1, offset: 5` in the request body still exports 16,031 rows. - `format=xlsx` → 400 with a JSON error, path outside the allowed directories → 403, missing file inside them → 404 (resolved before any bytes are written, so no truncated file). - Peak RSS stayed flat across 200 consecutive exports. `mvn test`: 297 run, 0 failures, 17 skipped (11 new). `pnpm build` passes. ## Not addressed here - The export does not apply `query.timeout-seconds`; a long export holds the request thread. Capping it belongs at the gateway. - Each page re-scans the file from the start, so a very large export is O(n² / pageSize) in scan work. `TsFileDataReader.streamData(...)` can make this a single pass in a follow-up. - `docs/API.md` and `docs/MANUAL_TEST_CASES.zh-CN.md` (TC-C005) still describe the old page-scoped behaviour. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
