[
https://issues.apache.org/jira/browse/AIRFLOW-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Potiuk closed AIRFLOW-2964.
---------------------------------
Resolution: Won't Fix
I am closing some old issues that are not relevant any more. Please let me know
if you want to reopen it.
> Lazy generation of the job description
> --------------------------------------
>
> Key: AIRFLOW-2964
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2964
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib, operators
> Affects Versions: 1.9.0, 2.0.0
> Reporter: Victor Jimenez
> Priority: Major
>
> When instantiating a {{DatabricksSubmitRunOperator}} users need to pass the
> description of the job that will later be executed on Databricks.
> The job description is only needed at execution time (when the hook is
> called). However, the {{json}} parameter must already have the full job
> description when constructing the operator. This may present a problem if
> computing the job description needs to execute expensive operations (e.g.,
> querying a database). The expensive operation will be invoked every single
> time the DAG is reprocessed (which may happen quite frequently).
> It would be good to have an equivalent mechanism to the {{python_callable}}
> parameter in the {{PythonOperator}}. In this way, users could pass a function
> that would generate the job description only when the operator is actually
> executed.
> [~andrewmchen] Is there any other way to do this? If not, does this sound
> reasonable? I can create a PR implementing this proposal.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)