aicam opened a new issue, #8445:
URL: https://github.com/apache/texera/issues/8445
### What happened?
Downloading a large file from a dataset through the UI **silently produces a
truncated file**. There is no error in the browser, no error toast in Texera,
and no error in the file-service logs — the browser reports the download as
*completed*, but the resulting file is only a small fraction of the real object.
Observed on a Kubernetes deployment (Helm chart in `bin/k8s`) with a dataset
containing multi-GB Zeiss `.czi` microscopy files:
| File | Size reported by `rootFileNodes` | Bytes actually downloaded | % |
|---|---|---|---|
| `A.czi` | 1,416,092,224 | 106,168,320 | 7.5% |
| `B.czi` | 2,424,775,680 | 92,546,841 | 3.8% |
| `<dataset>-v1.zip` (Download Version) | ~9.39 GB | 51,367,792 | 0.5% |
The downloaded files are structurally valid at the start — the `.czi` still
begins with the `ZISRAWFILE` magic bytes — they just end abruptly. The version
zip has no central directory (`unzip -t` → "cannot find zipfile directory"), so
it cannot be opened at all. Downstream tools (Bio-Formats / Fiji, `unzip`) fail
on the results.
Note that the truncation points are **not** a constant byte count and are
not round numbers. They correspond to a constant *wall-clock* window of roughly
15 seconds at the throughput each transfer happened to get (~7.1, ~6.2 and ~3.4
MB/s respectively).
**Root cause:** dataset file downloads are served by redirecting the browser
to a **presigned S3 URL pointing at the MinIO hostname**, and that hostname is
routed through Envoy Gateway by `texera-minio-route` in
`bin/k8s/templates/gateway-routes.yaml`:
```yaml
# MinIO Route
{{- if .Values.minio.gateway.enabled }}
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: texera-minio-route
...
spec:
...
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: {{ .Release.Name }}-minio
port: 9000
{{- end }}
```
This rule declares **no `timeouts` block**, so Envoy Gateway applies its
default HTTP route request timeout of **15s**. Any object that takes longer
than 15s to transfer gets cut off mid-stream. Because the presigned `GET`
response is streamed and the connection is then closed cleanly, the browser
does not treat it as a failed download — it just writes a short file. The
result is **silent data corruption** rather than a visible error.
`kubectl -n <ns> get httproute texera-minio-route -o yaml` on the live
cluster confirms there is no `spec.rules[].timeouts` field.
**Expected:** downloading a dataset file should yield the complete object
regardless of size/duration, or fail loudly if it cannot.
**Impact:** effectively every dataset file larger than ~15 seconds of
transfer time is silently unusable. For imaging/genomics datasets (the multi-GB
case Texera datasets are meant to hold) this means the download feature does
not work at all, and users get corrupted data without knowing it.
**Suggested fix:**
1. Set an explicit long (or disabled) timeout on the MinIO `HTTPRoute` in
`bin/k8s/templates/gateway-routes.yaml`, ideally configurable via `values.yaml`:
```yaml
timeouts:
request: 0s # disable; Envoy Gateway treats 0s as no timeout
backendRequest: 0s
```
The same should be reviewed for the `file-service` route, which serves
the `/{did}/versionZip` streaming endpoint and has the identical problem.
2. Independently of the gateway config, the frontend should verify the
downloaded byte count against the known object size and surface an error
instead of handing the user a truncated file.
### How to reproduce?
1. Deploy Texera on Kubernetes using the Helm chart in `bin/k8s` with
`minio.gateway.enabled=true` (i.e. presigned downloads served through Envoy
Gateway).
2. Create a dataset and upload a large file — ~2 GB is enough; anything
whose transfer takes more than ~15s will do.
3. Open the dataset page in the UI, select the file, and click the download
button.
4. Observe the browser reports the download as complete with no error.
5. Compare the size of the downloaded file with the size shown in the
dataset file list — the local file is a small fraction of it.
6. Same for "Download Version" (the zip): the resulting archive is truncated
and `unzip -t` reports a missing central directory.
### Version/Branch
1.3.0-incubating-SNAPSHOT (main)
### What browsers are you seeing the problem on?
Chrome
### Relevant log output
No error is logged — that is a core part of the problem. The file-service
reports a successful presign, MinIO reports a normal GET, and the browser
reports a completed download.
Evidence of the truncation, taken locally after the "successful" download:
$ stat -c%s '<file>.czi'
106168320 # dataset file list reports 1416092224
$ head -c 16 '<file>.czi' | xxd
00000000: 5a49 5352 4157 4649 4c45 0000 0000 0000 ZISRAWFILE......
# valid CZI header - the stream is real data, it
just stops early
$ unzip -t '<dataset>-v1.zip'
cannot find zipfile directory in one of <dataset>-v1.zip ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
