eye-gu commented on issue #18350: URL: https://github.com/apache/dolphinscheduler/issues/18350#issuecomment-4706025073
For small-to-medium clusters where Git concurrency is not an issue, real-time download (via the REST API) works fine. If concurrency does become a bottleneck, we can introduce a config switch that lets each worker maintain a local clone refreshed by periodic git fetch, so task downloads just copy local files instead of hitting Git. The trade-off is that users need to configure a sync interval, and within that interval resources may be inconsistent. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
