[GitHub] [airflow] jose-lpa commented on issue #29432: Jinja templating doesn't work with container_resources when using dymanic task mapping with Kubernetes Pod Operator

[via GitHub]

jose-lpa commented on issue #29432:
URL: https://github.com/apache/airflow/issues/29432#issuecomment-1437182956

   I actually fixed my situation by simply using the [`Variable` 
model](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/variables.html)
 instead of trying to go with templated stuff.
   
   Example of my exact situation:
   
   **Failing:**
   ```python
   with DAG(...) as dag:
   
       # ...other tasks...
   
       calculate_statistics = KubernetesPodOperator.partial(
           config_file="/home/airflow/composer_kube_config",
           kubernetes_conn_id="kubernetes_default",
           namespace="default",
           task_id="calculate_statistics",
           name="calculate-statistics",
           image=(
               "eu.gcr.io/hummingbird-technologies/tasks/imagery-stats:{{ 
var.value.ENVIRONMENT }}"
           ),
           image_pull_policy="Always",
           env_vars=[
               V1EnvVar(name="INTERNAL_API_URL", value="{{ 
var.value.INTERNAL_API_URL }}",
               V1EnvVar(name="COLLECTION_NAME", value="{{ 
var.value.COLLECTION_NAME }}"),
           ],
           container_resources=V1ResourceRequirements(requests={"cpu": 1, 
"memory": "10Gi"}),
           startup_timeout_seconds=5 * 60,
           retries=0,
       )
   
       statistics = 
calculate_statistics.expand(arguments=XComArg(argument_builder))
   
       chain(acquire_data, statistics)
   ```
   
   **Working:**
   ```python
   from airflow.models import Variable
   
   
   with DAG(...) as dag:
   
       # ...other tasks...
   
       calculate_statistics = KubernetesPodOperator.partial(
           config_file="/home/airflow/composer_kube_config",
           kubernetes_conn_id="kubernetes_default",
           namespace="default",
           task_id="calculate_statistics",
           name="calculate-statistics",
           image=(
               "eu.gcr.io/hummingbird-technologies/tasks/imagery-stats:{{ 
var.value.ENVIRONMENT }}"
           ),
           image_pull_policy="Always",
           env_vars=[
               V1EnvVar(name="INTERNAL_API_URL", 
value=Variable.get("INTERNAL_API_URL")),
               V1EnvVar(name="COLLECTION_NAME", 
value=Variable.get("COLLECTION_NAME")),
           ],
           container_resources=V1ResourceRequirements(requests={"cpu": 1, 
"memory": "10Gi"}),
           startup_timeout_seconds=5 * 60,
           retries=0,
       )
   
       statistics = 
calculate_statistics.expand(arguments=XComArg(argument_builder))
   
       chain(acquire_data, statistics)
   ```
   
   @vasu2809 maybe this can help you too...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org