CritasWang opened a new pull request, #51:
URL: https://github.com/apache/tsfile-viewer/pull/51

   Fixes #45
   
   ## Problem
   
   `exportCSV()` / `exportJSON()` serialized `dataRows`, which only holds the 
current preview page, so the exported subset changed whenever the user paged. 
The user guide promises the export covers every record matching the filters.
   
   Paging the preview endpoint from the browser is not enough: with post-read 
filters the backend caps a single read at `query.max-result-size` (10,000), so 
the export still truncated there — measured 10,000 rows on a 16,031-row file.
   
   ## Changes
   
   - New `POST /api/data/export?format=csv|json` streams the whole filtered 
result set. The server pages through the file itself (2,000 rows per page, 
offset pushed down), applies exactly the same filters as the preview, and 
flushes after every page, so the file never has to fit in memory and 
`max-result-size` no longer truncates. `limit`/`offset` in the request body are 
ignored.
   - CSV follows RFC 4180 (commas, quotes, line breaks quoted and escaped), 
columns are the file's column union in lexicographic order, and timestamps are 
ISO-8601 UTC with milliseconds — the same text the browser used to produce.
   - JSON streams one row object per line, identical in shape to the preview 
rows.
   - Frontend requests the blob from the new endpoint and keeps the export 
buttons disabled while it runs.
   - `DataPreviewRequest` now accepts `advancedConditions`: the frontend has 
been sending it, but the backend silently dropped it, so advanced filters never 
applied to the preview either.
   
   ## Verification
   
   Against a 16,031-row tree-model file in a local instance, driven through the 
real UI with Playwright:
   
   | | before | after |
   |---|---|---|
   | rows on the preview page | 100 | 100 |
   | CSV data rows | 100 | **16,031** |
   | JSON rows | 100 | **16,031** |
   
   - ~100 ms for either format including the download; no console errors.
   - `limit: 1, offset: 5` in the request body still exports 16,031 rows.
   - `format=xlsx` → 400 with a JSON error, path outside the allowed 
directories → 403, missing file inside them → 404 (resolved before any bytes 
are written, so no truncated file).
   - Peak RSS stayed flat across 200 consecutive exports.
   
   `mvn test`: 297 run, 0 failures, 17 skipped (11 new). `pnpm build` passes.
   
   ## Not addressed here
   
   - The export does not apply `query.timeout-seconds`; a long export holds the 
request thread. Capping it belongs at the gateway.
   - Each page re-scans the file from the start, so a very large export is O(n² 
/ pageSize) in scan work. `TsFileDataReader.streamData(...)` can make this a 
single pass in a follow-up.
   - `docs/API.md` and `docs/MANUAL_TEST_CASES.zh-CN.md` (TC-C005) still 
describe the old page-scoped behaviour.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to