[
https://issues.apache.org/jira/browse/AIRFLOW-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959101#comment-15959101
]
ASF subversion and git services commented on AIRFLOW-1028:
----------------------------------------------------------
Commit 53ca5084561fd5c13996609f2eda6baf717249b5 in incubator-airflow's branch
refs/heads/master from [~andrewmchen]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=53ca508 ]
[AIRFLOW-1028] Databricks Operator for Airflow
Add DatabricksSubmitRun Operator
In this PR, we contribute a DatabricksSubmitRun operator and a
Databricks hook. This operator enables easy integration of Airflow
with Databricks. In addition to the operator, we have created a
databricks_default connection, an example_dag using this
DatabricksSubmitRunOperator, and matching documentation.
Closes #2202 from andrewmchen/databricks-operator-
squashed
> Databricks Operator for Airflow
> -------------------------------
>
> Key: AIRFLOW-1028
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1028
> Project: Apache Airflow
> Issue Type: New Feature
> Reporter: Andrew Chen
> Assignee: Andrew Chen
>
> It would be nice to have a Databricks Operator/Hook in Airflow so users of
> Databricks can more easily integrate with Airflow.
> The operator would submit a spark job to our new /jobs/runs/submit endpoint.
> This endpoint is similar to
> https://docs.databricks.com/api/latest/jobs.html#jobscreatejob but does not
> include the email_notifications, max_retries, min_retry_interval_millis,
> retry_on_timeout, schedule, max_concurrent_runs fields. (The submit docs are
> not out because it's still a private endpoint.)
> Our proposed design for the operator then is to match this REST API endpoint.
> Each argument to the parameter is named to be one of the fields of the REST
> API request and the value of the argument will match the type expected by the
> REST API. We will also merge extra keys from kwargs which should not be
> passed to the BaseOperator into our API call in order to be flexible to
> updates.
> In the case that this interface is not very user friendly, we can later add
> more operators which extend this operator.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
