CritasWang opened a new pull request, #50:
URL: https://github.com/apache/tsfile-viewer/pull/50

   ## Problem
   
   Two memory blow-ups with one root cause: per-chunk data is materialised even 
though it is never needed.
   
   1. `GET /api/meta/{fileId}` returned every row group **and every chunk** of 
the file. For a real 87,488-row-group file that is ~1.8M chunk DTOs in a single 
JSON response. Parsing it exhausted the 2 GB heap; the resulting 
`OutOfMemoryError` killed Tomcat's poller thread while the acceptor kept 
accepting connections, so the service stayed "up" for 7 days answering every 
request with zero bytes.
   2. `TsFileHealthChecker` built a `HashMap<Long, IChunkMetadata>` entry for 
every chunk, so scanning the same file had the same failure mode.
   
   ## Changes
   
   **Metadata**
   - `GET /api/meta/{fileId}` returns a summary only; `rowGroups` / `chunks` 
remain in the contract but are always empty arrays.
   - New `GET /api/meta/{fileId}/row-groups` and `/chunks` with 
`offset`/`limit` (default 50, max 200), returning 
`{items,total,offset,limit,hasMore}`.
   - Reads are per-device and incremental — no full 
`getAllTimeseriesMetadata(true)` materialisation on the metadata path.
   - Metadata cache holds the summary only; a per-file device index is cached 
and validated by `mtime` + `size`.
   - Frontend: `el-table-v2` virtualised rows with load-on-scroll for both 
tables (the chunk table did not exist before). Device options come from a 
bounded paged scan and are labelled "(loaded / total)" rather than pretending 
to be complete.
   - `el-table-v2` styles are imported explicitly — components imported in 
`<script setup>` do not get CSS injected by `unplugin-vue-components`, and 
without it the header lands below the body and rows overlap.
   
   **Scan**
   - New `ChunkMetadataIndex`: parallel primitive arrays (offset / measurement 
id / start / end), 28 B per chunk ≈ 49 MiB for 1.83M chunks versus roughly 350 
MiB+ for the object map. Forward-only cursor with binary-search fallback.
   - `quickCheck` stops at the first chunk instead of building the whole index.
   - New `tsfile.scan.max-chunks` (default 5,000,000): above the cap the scan 
degrades to structure + metadata checks and reports `DATA_VALIDATION_SKIPPED` 
(WARNING) instead of OOMing.
   
   **Dependency / config**
   - tsfile 2.3.0 → 2.5.0.
   - Application log level DEBUG → INFO (`logging.level` used a package name 
that does not exist, so the configured INFO never took effect).
   
   ## Verification
   
   - `mvn test`: 322 run, 0 failures, 17 skipped. `pnpm build` passes (vue-tsc 
included).
   - Real 729 MB / 1,838,571-chunk file, throwaway instance with `-Xmx512m`: 
scan completes in 30 s, `HEALTHY`, no OOM. With `--tsfile.scan.max-chunks=1000` 
it degrades in 138 ms to `WARNING` + `DATA_VALIDATION_SKIPPED`.
   - Live summary for that file: cold 6.9 s (device index build), warm 6.7 ms; 
live heap after GC 44 MB.
   - Playwright against the deployed build: both tables render with a correct 
header, rows do not overlap, counters advance on scroll (`50/87488 → 
450/87488`, `50/1838571 → 450/1838571`), zero console errors.
   
   ## Note
   
   Unrelated pre-existing issue found while testing: tree-model files fail data 
preview and chart with a `NullPointerException` inside 
`DeviceTableModelReader.getAllTableSchema` (tsfile). Reproduced on the 
pre-upgrade 2.3.0 build as well, so it is not introduced here and is not 
addressed in this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to