jingi723 commented on issue #72129: URL: https://github.com/apache/airflow/issues/72129#issuecomment-6080645534
I have a local read-only candidate: async deserialize hooks preserve the existing instance/thread for sync-only backends, while the object-storage backend provides native reads for an already initialized standard s3fs filesystem with plain/gzip objects. I kept the default hooks on the caller's thread because a thread-bound SQLite custom backend failed under automatic thread offloading. Writes and blocking legacy fallbacks remain unchanged. With s3fs 2026.9.0 and 75 ms of injected local HTTP latency, ten concurrent task reads completed in a median 940 → 124 ms. SDK/provider HTTP checks cover input lineage on errors, conditional reads when an object changes, and response-body cleanup on cancellation. Focused Airflow timeout/supervisor checks also completed without forced termination. These are controlled local results, not live S3 measurements or the full acceptance matrix. Cold filesystem/credential initialization is still unresolved by this change. One design choice remains: preserving the S3 read contract currently makes common.io depend on the private `s3fs.core._inner_fetch` helper. Would you consider the async hooks and this narrowly scoped reader together, or prefer a public async read abstraction in the filesystem layer first? Is there an existing implementation I should coordinate with? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
