Donniedarko45 opened a new pull request, #76915:
URL: https://github.com/apache/airflow/pull/76915
`SparkKubernetesOperator` fails when the memory limit in
`container_resources` is written as `512Mi`, `512M` or `2g`:
TypeError: unsupported operand type(s) for /: 'str' and 'float'
`SparkResources.convert_resources` only converted limits written with `m`,
`G` or `Gi`. Anything else was left as a string and then divided by 1.4, which
gives the error above and no hint about what is wrong.
This accepts `m`/`M`/`Mi` and `g`/`G`/`Gi` in any case. `M`/`Mi` are treated
the same as `m`, the same way `G` and `Gi` were already treated alike. Limits
that already worked give the same result as before. A limit in a unit that
still can't be converted (for example `1Ti`, or a bare number) now raises a
`ValueError` that names the value and the supported units, instead of the
`TypeError`. The driver and executor used two copies of the same code, so both
now go through one helper.
Tests: added cases for the three spellings that used to fail and for the
unsupported-unit error. They fail without the change.
`test_custom_object_launcher.py` and `test_spark_kubernetes.py` pass.
Pre-commit checks pass except `check-template-fields-valid`, which needs breeze
and Docker and was not run locally (same for provider mypy).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
