amoghrajesh commented on PR #37214: URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931650831
Awesome. I was able to test it. Results: My connections defined: <img width="867" alt="image" src="https://github.com/apache/airflow/assets/35884252/06a85bbd-579b-436b-9079-ec27499120c2";> Logs for my_mong (ssl is not present): ``` c47a4cad3064 *** Found local files: *** * /root/airflow/logs/dag_id=mongo_dag/run_id=manual__2024-02-07T09:32:50.030512+00:00/task_id=retrieve_users_from_mongodb/attempt=1.log [2024-02-07, 15:02:53 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=non-requeueable deps ti= [2024-02-07, 15:02:53 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=requeueable deps ti= [2024-02-07, 15:02:53 IST] {taskinstance.py:2194} INFO - Starting attempt 1 of 1 [2024-02-07, 15:02:53 IST] {taskinstance.py:2215} INFO - Executing on 2024-02-07 09:32:50.030512+00:00 [2024-02-07, 15:02:53 IST] {standard_task_runner.py:60} INFO - Started process 1073 to run task [2024-02-07, 15:02:53 IST] {standard_task_runner.py:87} INFO - Running: ['airflow', 'tasks', 'run', 'mongo_dag', 'retrieve_users_from_mongodb', 'manual__2024-02-07T09:32:50.030512+00:00', '--job-id', '929', '--raw', '--subdir', 'DAGS_FOLDER/mongo_test.py', '--cfg-path', '/tmp/tmp1dqi4wjk'] [2024-02-07, 15:02:53 IST] {standard_task_runner.py:88} INFO - Job 929: Subtask retrieve_users_from_mongodb [2024-02-07, 15:02:53 IST] {task_command.py:423} INFO - Running on host c47a4cad3064 [2024-02-07, 15:02:53 IST] {taskinstance.py:2515} INFO - Exporting env vars: AIRFLOW_CTX_DAG_EMAIL='[email protected]' AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='mongo_dag' AIRFLOW_CTX_TASK_ID='retrieve_users_from_mongodb' AIRFLOW_CTX_EXECUTION_DATE='2024-02-07T09:32:50.030512+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-02-07T09:32:50.030512+00:00' [2024-02-07, 15:02:53 IST] {base.py:83} INFO - Using connection ID 'my_mong' for task execution. [2024-02-07, 15:02:53 IST] {logging_mixin.py:188} INFO - {'_id': ObjectId('65c34e0e7e0c274d333d8922'), 'username': 'user1', 'email': '[email protected]'} [2024-02-07, 15:02:53 IST] {logging_mixin.py:188} INFO - {'_id': ObjectId('65c34e0e7e0c274d333d8923'), 'username': 'user2', 'email': '[email protected]'} [2024-02-07, 15:02:53 IST] {python.py:201} INFO - Done. Returned value was: None [2024-02-07, 15:02:53 IST] {taskinstance.py:1152} INFO - Marking task as SUCCESS. dag_id=mongo_dag, task_id=retrieve_users_from_mongodb, execution_date=20240207T093250, start_date=20240207T093253, end_date=20240207T093253 [2024-02-07, 15:02:53 IST] {local_task_job_runner.py:234} INFO - Task exited with return code 0 [2024-02-07, 15:02:53 IST] {taskinstance.py:3318} INFO - 0 downstream tasks scheduled from follow-on schedule check ``` Logs for my_mong_secure (explicitly passed my ssl: true) ``` c47a4cad3064 *** Found local files: *** * /root/airflow/logs/dag_id=mongo_dag/run_id=manual__2024-02-07T09:36:02.449095+00:00/task_id=retrieve_users_from_mongodb/attempt=1.log [2024-02-07, 15:06:05 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=non-requeueable deps ti= [2024-02-07, 15:06:05 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=requeueable deps ti= [2024-02-07, 15:06:05 IST] {taskinstance.py:2194} INFO - Starting attempt 1 of 1 [2024-02-07, 15:06:05 IST] {taskinstance.py:2215} INFO - Executing on 2024-02-07 09:36:02.449095+00:00 [2024-02-07, 15:06:05 IST] {standard_task_runner.py:60} INFO - Started process 1397 to run task [2024-02-07, 15:06:05 IST] {standard_task_runner.py:87} INFO - Running: ['airflow', 'tasks', 'run', 'mongo_dag', 'retrieve_users_from_mongodb', 'manual__2024-02-07T09:36:02.449095+00:00', '--job-id', '930', '--raw', '--subdir', 'DAGS_FOLDER/mongo_test.py', '--cfg-path', '/tmp/tmpn18c8wnm'] [2024-02-07, 15:06:05 IST] {standard_task_runner.py:88} INFO - Job 930: Subtask retrieve_users_from_mongodb [2024-02-07, 15:06:05 IST] {task_command.py:423} INFO - Running on host c47a4cad3064 [2024-02-07, 15:06:05 IST] {taskinstance.py:2515} INFO - Exporting env vars: AIRFLOW_CTX_DAG_EMAIL='[email protected]' AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='mongo_dag' AIRFLOW_CTX_TASK_ID='retrieve_users_from_mongodb' AIRFLOW_CTX_EXECUTION_DATE='2024-02-07T09:36:02.449095+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-02-07T09:36:02.449095+00:00' [2024-02-07, 15:06:05 IST] {base.py:83} INFO - Using connection ID 'my_mong_secure' for task execution. [2024-02-07, 15:06:35 IST] {taskinstance.py:2737} ERROR - Task failed with exception Traceback (most recent call last): File "/opt/airflow/airflow/models/taskinstance.py", line 446, in _execute_task result = _execute_callable(context=context, **execute_callable_kwargs) File "/opt/airflow/airflow/models/taskinstance.py", line 416, in _execute_callable return execute_callable(context=context, **execute_callable_kwargs) File "/opt/airflow/airflow/operators/python.py", line 199, in execute return_value = self.execute_callable() File "/opt/airflow/airflow/operators/python.py", line 216, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/files/dags/mongo_test.py", line 28, in retrieve_users_from_mongodb for user in users_data: File "/usr/local/lib/python3.8/site-packages/pymongo/cursor.py", line 1264, in next if len(self.__data) or self._refresh(): File "/usr/local/lib/python3.8/site-packages/pymongo/cursor.py", line 1155, in _refresh self.__session = self.__collection.database.client._ensure_session() File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1823, in _ensure_session return self.__start_session(True, causal_consistency=False) File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1766, in __start_session self._topology._check_implicit_session_support() File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 573, in _check_implicit_session_support self._check_session_support() File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 589, in _check_session_support self._select_servers_loop( File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 259, in _select_servers_loop raise ServerSelectionTimeoutError( pymongo.errors.ServerSelectionTimeoutError: SSL handshake failed: 172.20.0.3:27017: TLS/SSL connection has been closed (EOF) (_ssl.c:1131) (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 30s, Topology Description: ]> [2024-02-07, 15:06:35 IST] {taskinstance.py:1152} INFO - Marking task as FAILED. dag_id=mongo_dag, task_id=retrieve_users_from_mongodb, execution_date=20240207T093602, start_date=20240207T093605, end_date=20240207T093635 [2024-02-07, 15:06:35 IST] {standard_task_runner.py:107} ERROR - Failed to execute job 930 for task retrieve_users_from_mongodb (SSL handshake failed: 172.20.0.3:27017: TLS/SSL connection has been closed (EOF) (_ssl.c:1131) (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 30s, Topology Description: ]>; 1397) [2024-02-07, 15:06:35 IST] {local_task_job_runner.py:234} INFO - Task exited with return code 1 [2024-02-07, 15:06:35 IST] {taskinstance.py:3318} INFO - 0 downstream tasks scheduled from follow-on schedule check ``` cc @Taragolis @potiuk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
