loserwang1024 commented on PR #654:
URL: https://github.com/apache/fluss-rust/pull/654#issuecomment-4934216485
@fresh-borzoni
Thank you very much for the thorough review. You were absolutely right that
schema prewarming only covered the local path, while tiered/remote data could
still fail with `No ReadContext found for schema_id N`.
I explored several approaches because the remote path behaves differently
from the local one. Local records are included directly in the fetch response,
so their schema IDs can be collected and prewarmed before decoding. For tiered
data, however, the fetch response only contains remote segment metadata. The
actual file must first be downloaded and is then read lazily, batch by batch.
Discovering all schema IDs in advance would require scanning the entire
downloaded segment before returning any data, which would defeat the
file-backed streaming behavior and increase both latency and memory usage.
My first attempt in commit `04eb30dc` therefore fetched the schema lazily
when the concrete batch was being decoded. However, schema resolution was still
invoked from a synchronous decoding path. It used a Tokio runtime handle from
another thread and then synchronously joined that thread. On a current-thread
Tokio runtime, this could deadlock because the blocked runtime thread was also
required to drive the schema RPC.
The latest fix keeps the per-batch lazy-resolution approach but moves the
asynchronous work to the scanner layer:
- The synchronous decoder returns `SchemaRequired(schema_id)` when it
encounters an uncached schema.
- The current raw batch is retained instead of being discarded or advancing
through the remote file.
- The asynchronous scanner fetches and registers the schema without
blocking the Tokio runtime.
- The same batch is then retried and decoded.
This allows both local and remote paths to resolve schemas on demand
without preloading the entire remote segment and without blocking the runtime.
I also added coverage for both dynamic-schema and fixed-schema modes.
Thanks again for catching this and for the very detailed review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
