mpolatcan commented on issue #33688:
URL: https://github.com/apache/airflow/issues/33688#issuecomment-1696254990
Hi again @potiuk, I didn't mention anything about pickle library @jaetma
mentioned it ☺️ And also, our environment runs custom packaged Docker image
that basically install main dependencies and apache-airflow[all]==2.7.0 package
with constraints file. We packaged Docker image with python:3.9 official Python
image.
```python
from airflow.models import DAG
from datetime import datetime, timedelta
from utils import AirflowCallbacks
from operators import GetirKubernetesCronJobOperator
default_args = {
"owner": "Getir",
"catchup": False,
"depends_on_past": False,
"retries": 1,
"retry_delay": timedelta(seconds=15),
"start_date": datetime(year=2019, month=7, day=10)
}
dag = DAG(
dag_id="client_list",
default_args=default_args,
schedule_interval=None,
tags=["client-list", "reports"]
)
GetirKubernetesCronJobOperator(
name="cl-",
task_id="clientlist",
namespace="airflow",
service_account_name="airflow-sa-role-sa",
image="164762854291.dkr.ecr.eu-west-1.amazonaws.com/getir-python:3.7-buster",
image_pull_policy="IfNotPresent",
repo_name="client-list-cron",
exec_env="eks",
eks_nodegroup="ng-airflow-reports",
eks_cpu_limit="500m",
eks_memory_limit="16000Mi",
cmds=[
"bash", "-c",
(
"mkdir -p /usr/src/app && cp -r /repo/* /usr/src/app && "
"cd /usr/src/app && pip install -r requirements.txt -q && "
"python -u main.py {{ dag_run.conf.get('parameters') }}"
)
],
labels={"report": "yes"},
annotations={
"report_type": "panel",
"report_name": "client-list",
"exec_env": "{{dag_run.conf.get('exec_env', 'fargate')}}",
},
startup_timeout_seconds=600,
is_delete_operator_pod=False,
reattach_on_restart=False,
pool="client_list_pool",
on_failure_callback=AirflowCallbacks.client_list_failed_callback,
dag=dag
)
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
