GitHub user starkmarkus added a comment to the discussion: storage for tracking rate limits
No, this is separate from the task-execution broker. The rate limiter backend only stores short-lived counters for the limiter itself, not the task queue or other Airflow internals. It does not handle metrics. It is just shared storage for rate-limit state. On Valkey vs Redis: Valkey is a Redis fork and is generally used as a Redis-compatible replacement for common use cases. If you already have Redis, that is the safest default. If your setup already uses Valkey, it should usually work too, but I would still test it in your exact deployment. And yes, having multiple webserver/API processes is normal in production. The important part is that they share the same limiter backend, otherwise each process would track limits independently. GitHub link: https://github.com/apache/airflow/discussions/68630#discussioncomment-17329102 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
