potiuk commented on a change in pull request #6740: [AIRFLOW-6181] Add InProcessExecutor URL: https://github.com/apache/airflow/pull/6740#discussion_r355671817
########## File path: airflow/executors/inprocess_executor.py ########## @@ -0,0 +1,152 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +This module contains InProcessExecutor that is a single +process executor meaning it does not use multiprocessing. +""" + +import threading +from typing import Any, Dict, List, Optional + +from airflow import conf +from airflow.executors.base_executor import BaseExecutor +from airflow.models.taskinstance import TaskInstance, TaskInstanceKeyType +from airflow.utils.state import State + + +class InProcessExecutor(BaseExecutor): + """ + This executor is meant for debugging purposes. It can be used with + sqlite since sqlite do not support multiple connections. + + It executes one task instance at time. Additionally to support working + with sensors, sensor's ``mode`` will be automatically set to "reschedule". Review comment: The problem is that if sensor depends on another task doing it's work, it will block the executor and will not let the operator to do its job. We had a few examples of those when we worked on GCP operators. Basically - when you want to spin-off both sensor and operator at the same time you need this. Example here: https://github.com/apache/airflow/blob/master/airflow/gcp/example_dags/example_bigtable.py Here we have two sensors waiting and they might be fired in any order - they are waiting for two creates (they trigger the sensor's replicate wait to complete). We have more than one sensor and they can fire in any order as they are not depending on each other. Using "reschedule" mode makes it robust to more strict dependencies set in the DAG. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
