jscheffl commented on PR #42410: URL: https://github.com/apache/airflow/pull/42410#issuecomment-2453088900
> > I don't see a point that there is a real "functional need" to set priorities exponential or in another manner that - with a normal modelling - yo need such large ranges > > Let's not forget that we have the priority-weight cumulation: upstream, downstream and eventually Custom: https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/priority-weight.html#custom-weight-rule > > I can quite easily imagine some big numbers when we have huge 1000s task dags with multiple upstream/downstream tasks (and layers) and especially with custom weights I can easily imagine those numbers to add up (or maybe multiply in custom rules if one wants to have more aggressive priority propagation). I still don't see a real need and use of such high numbers. Yes we accumulate priority weights by making sums. Assume we have a DAG with 1000 tasks chained (I hope nobody is modelling this, will really run a long time) and we use a priority of 10k (=10000). Then the accumulated priority is at 10 million. Looking into the INT value we use today the supported database have integer ranges with: - postgres: -2147483648 to +2147483647 (see https://www.postgresql.org/docs/current/datatype-numeric.html) - mysql: -2147483648 to +2147483647 (see https://dev.mysql.com/doc/refman/8.4/en/integer-types.html) - sqlite: -9223372036854775808 to +9223372036854775807 (see https://sqlite.org/datatype3.html) This means I still can have 1000 tasks with a priority of 1 million in my DAG. Which is also something in the range of values you can model to fit into the INT range. Instead of switching to float I think we should rather cap the values and ensure they can not roll-over. And add documentation about the limits. The limtis of postgres and mysql are the same and sound reasonable (else: there is also the option to switch to bigint of course if you want to support incredible numbers non-float). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
