[ https://issues.apache.org/jira/browse/AIRFLOW-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959101#comment-15959101 ]
ASF subversion and git services commented on AIRFLOW-1028: ---------------------------------------------------------- Commit 53ca5084561fd5c13996609f2eda6baf717249b5 in incubator-airflow's branch refs/heads/master from [~andrewmchen] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=53ca508 ] [AIRFLOW-1028] Databricks Operator for Airflow Add DatabricksSubmitRun Operator In this PR, we contribute a DatabricksSubmitRun operator and a Databricks hook. This operator enables easy integration of Airflow with Databricks. In addition to the operator, we have created a databricks_default connection, an example_dag using this DatabricksSubmitRunOperator, and matching documentation. Closes #2202 from andrewmchen/databricks-operator- squashed > Databricks Operator for Airflow > ------------------------------- > > Key: AIRFLOW-1028 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1028 > Project: Apache Airflow > Issue Type: New Feature > Reporter: Andrew Chen > Assignee: Andrew Chen > > It would be nice to have a Databricks Operator/Hook in Airflow so users of > Databricks can more easily integrate with Airflow. > The operator would submit a spark job to our new /jobs/runs/submit endpoint. > This endpoint is similar to > https://docs.databricks.com/api/latest/jobs.html#jobscreatejob but does not > include the email_notifications, max_retries, min_retry_interval_millis, > retry_on_timeout, schedule, max_concurrent_runs fields. (The submit docs are > not out because it's still a private endpoint.) > Our proposed design for the operator then is to match this REST API endpoint. > Each argument to the parameter is named to be one of the fields of the REST > API request and the value of the argument will match the type expected by the > REST API. We will also merge extra keys from kwargs which should not be > passed to the BaseOperator into our API call in order to be flexible to > updates. > In the case that this interface is not very user friendly, we can later add > more operators which extend this operator. -- This message was sent by Atlassian JIRA (v6.3.15#6346)