kacpermuda commented on code in PR #75678:
URL: https://github.com/apache/airflow/pull/75678#discussion_r4403411708
##########
airflow-e2e-tests/docker/openlineage.yml:
##########
@@ -35,10 +35,86 @@ x-ol-common-env: &ol-common-env
# fully-qualified class path; the OpenLineage client imports it directly.
AIRFLOW__OPENLINEAGE__TRANSPORT: '{"type":
"system.openlineage.transport.variable.VariableTransport"}'
AIRFLOW__OPENLINEAGE__DISABLED: 'false'
- # Deployment-wide policy required by the two conf-driven policy DAGs. Each
rule is scoped by
- # dag_id, so both rules coexist and do not affect any other DAG.
- AIRFLOW__OPENLINEAGE__EMISSION_POLICY: '[{"scope": {"dag_id":
"openlineage_policy_conf_locked_dag"}, "locked": true, "controls":
{"include_source_code": false}}, {"scope": {"dag_id":
"openlineage_policy_conf_source_code_dag"}, "controls": {"include_source_code":
false}}]'
+ # Deployment-wide policy required by the conf-driven policy DAGs. Rules are
scoped by dag_id or by
+ # names unique to a DAG (hook/operator classes, dataset patterns), so they
coexist and do not affect any other DAG.
+ AIRFLOW__OPENLINEAGE__EMISSION_POLICY: >-
+ [
+ {"scope": {"dag_id": "openlineage_policy_conf_locked_dag"}, "locked":
true, "controls": {"include_source_code": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_source_code_dag"},
"controls": {"include_source_code": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_extract_off"}, "controls": {"extract_operator_metadata": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_hook_off"}, "controls": {"hook_lineage": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_full_info"}, "controls": {"include_full_task_info": true}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_emit_off"}, "controls": {"emit": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_task_events_off"}, "controls": {"emit_task_events": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_task_events_beat_emit"}, "controls": {"emit": false, "emit_task_events":
true}},
+ {"scope": {"operator": ".*\\.PolicyConfControlsOperator"}, "match_mode":
"regex", "controls": {"hook_lineage": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_operator_task"}, "controls": {"hook_lineage": true}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_extract_overridden"}, "controls": {"extract_operator_metadata": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_hook_overridden"}, "controls": {"hook_lineage": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_emit_overridden"}, "controls": {"emit": false}},
+ {"scope": {"operator": ".*\\.PolicyConfEmitOffOperator"}, "match_mode":
"regex", "controls": {"emit": false}},
+ {"scope": {"operator":
"airflow.providers.standard.operators.smooth.SmoothOperator"}, "controls":
{"emit": false}},
Review Comment:
Added the warning. It’s in the comment block above
AIRFLOW__OPENLINEAGE__EMISSION_POLICY, not right above the rule, because the
rules are inside a YAML >- string and a # line there would end up in the JSON.
##########
airflow-e2e-tests/docker/openlineage.yml:
##########
@@ -35,10 +35,86 @@ x-ol-common-env: &ol-common-env
# fully-qualified class path; the OpenLineage client imports it directly.
AIRFLOW__OPENLINEAGE__TRANSPORT: '{"type":
"system.openlineage.transport.variable.VariableTransport"}'
AIRFLOW__OPENLINEAGE__DISABLED: 'false'
- # Deployment-wide policy required by the two conf-driven policy DAGs. Each
rule is scoped by
- # dag_id, so both rules coexist and do not affect any other DAG.
- AIRFLOW__OPENLINEAGE__EMISSION_POLICY: '[{"scope": {"dag_id":
"openlineage_policy_conf_locked_dag"}, "locked": true, "controls":
{"include_source_code": false}}, {"scope": {"dag_id":
"openlineage_policy_conf_source_code_dag"}, "controls": {"include_source_code":
false}}]'
+ # Deployment-wide policy required by the conf-driven policy DAGs. Rules are
scoped by dag_id or by
+ # names unique to a DAG (hook/operator classes, dataset patterns), so they
coexist and do not affect any other DAG.
+ AIRFLOW__OPENLINEAGE__EMISSION_POLICY: >-
+ [
+ {"scope": {"dag_id": "openlineage_policy_conf_locked_dag"}, "locked":
true, "controls": {"include_source_code": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_source_code_dag"},
"controls": {"include_source_code": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_extract_off"}, "controls": {"extract_operator_metadata": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_hook_off"}, "controls": {"hook_lineage": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_full_info"}, "controls": {"include_full_task_info": true}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_emit_off"}, "controls": {"emit": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_task_events_off"}, "controls": {"emit_task_events": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_task_events_beat_emit"}, "controls": {"emit": false, "emit_task_events":
true}},
+ {"scope": {"operator": ".*\\.PolicyConfControlsOperator"}, "match_mode":
"regex", "controls": {"hook_lineage": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_operator_task"}, "controls": {"hook_lineage": true}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_extract_overridden"}, "controls": {"extract_operator_metadata": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_hook_overridden"}, "controls": {"hook_lineage": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_emit_overridden"}, "controls": {"emit": false}},
+ {"scope": {"operator": ".*\\.PolicyConfEmitOffOperator"}, "match_mode":
"regex", "controls": {"emit": false}},
+ {"scope": {"operator":
"airflow.providers.standard.operators.smooth.SmoothOperator"}, "controls":
{"emit": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"grp.t_grouped"}, "controls": {"hook_lineage": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_mapped"}, "controls": {"hook_lineage": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_contr.*", "task_id":
"t_regex_.*"}, "match_mode": "regex", "controls": {"hook_lineage": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_conflict"}, "controls": {"hook_lineage": true}},
+ {"scope": {"dag_id": "openlineage_policy_conf_controls_dag", "task_id":
"t_conflict"}, "controls": {"hook_lineage": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_locked_controls_dag",
"task_id": "t_locked_emit"}, "locked": true, "controls": {"emit": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_locked_controls_dag",
"task_id": "t_locked_task_events"}, "locked": true, "controls":
{"emit_task_events": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_locked_controls_dag",
"task_id": "t_locked_extract"}, "locked": true, "controls":
{"extract_operator_metadata": false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_locked_controls_dag",
"task_id": "t_locked_hook"}, "locked": true, "controls": {"hook_lineage":
false}},
+ {"scope": {"dag_id": "openlineage_policy_conf_locked_controls_dag",
"task_id": "t_locked_full"}, "locked": true, "controls":
{"include_full_task_info": true}},
+ {"scope": {"dag_id": "openlineage_policy_conf_locked_controls_dag"},
"locked": true, "controls": {"include_source_code": false, "exclude_datasets":
[".*/dag_locked_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_conf_locked_controls_dag",
"task_id": "t_floor"}, "controls": {"include_source_code": true}},
+ {"scope": {"dag_id":
"openlineage_policy_dag_events_off_ok_child__notrigger"}, "controls":
{"emit_dag_events": false}},
+ {"scope": {"dag_id":
"openlineage_policy_dag_events_off_fail_child__notrigger"}, "controls":
{"emit_dag_events": false}},
+ {"scope": {"dag_id":
"openlineage_policy_dag_events_on_emit_off_ok_child__notrigger"}, "controls":
{"emit": false, "emit_dag_events": true}},
+ {"scope": {"dag_id":
"openlineage_policy_dag_events_on_emit_off_fail_child__notrigger"}, "controls":
{"emit": false, "emit_dag_events": true}},
+ {"scope": {"dag_id":
"openlineage_policy_dag_events_emit_off_child__notrigger"}, "controls":
{"emit": false}},
+ {"scope": {"dag_id":
"openlineage_policy_dag_events_task_events_on_emit_off_child__notrigger"},
"controls": {"emit": false, "emit_task_events": true}},
+ {"scope": {"dag_id":
"openlineage_policy_dag_events_task_events_off_child__notrigger"}, "controls":
{"emit_task_events": false}},
+ {"scope": {"dag_id":
"openlineage_policy_dag_events_locked_off_child__notrigger"}, "locked": true,
"controls": {"emit_dag_events": false}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_authoring_dag"},
"controls": {"exclude_datasets": [".*/conf_excluded_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_authoring_dag",
"hook": ".*\\.AuthHook"}, "match_mode": "regex", "controls":
{"exclude_datasets": [".*/conf_hook_excluded_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_authoring_dag",
"task_id": "t_locked"}, "locked": true, "controls": {"exclude_datasets":
[".*/locked_excluded_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_authoring_dag",
"task_id": "t_locked", "hook": ".*\\.AuthHook"}, "match_mode": "regex",
"controls": {"exclude_datasets": [".*/specific_excluded_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_authoring_dag",
"task_id": "t_locked_hook", "hook": ".*\\.LockedHook"}, "match_mode": "regex",
"locked": true, "controls": {"exclude_datasets":
[".*/locked_hook_excluded_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_authoring_dag",
"hook": ".*\\.AuthDroppedHook"}, "match_mode": "regex", "controls":
{"hook_lineage": false}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_authoring_dag",
"task_id": "t_locked_dropped", "hook": ".*\\.AuthLockedDroppedHook"},
"match_mode": "regex", "locked": true, "controls": {"hook_lineage": false}},
+ {"scope": {}, "controls": {"exclude_datasets":
[".*/exclude_datasets_global_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_conf_dag"},
"controls": {"exclude_datasets": [".*/dag_excluded_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_conf_dag",
"hook": ".*\\.ConfHook"}, "match_mode": "regex", "controls":
{"exclude_datasets": [".*/dag_hook_excluded_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_conf_dag",
"task_id": "t_task"}, "controls": {"exclude_datasets":
[".*/task_excluded_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_conf_dag",
"task_id": "t_task_hook"}, "controls": {"exclude_datasets":
[".*/task_excluded_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_conf_dag",
"task_id": "t_task_hook", "hook": ".*\\.ConfHook"}, "match_mode": "regex",
"controls": {"exclude_datasets": [".*/task_hook_excluded_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_conf_dag",
"task_id": "t_task_clear"}, "controls": {"exclude_datasets": []}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_conf_dag",
"hook": ".*\\.DroppedHook"}, "match_mode": "regex", "controls":
{"hook_lineage": false}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_conf_dag",
"hook": ".*\\.KeptHook"}, "match_mode": "regex", "controls": {"hook_lineage":
true}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_conf_dag",
"task_id": "t_task_dropped", "hook": ".*\\.TaskDroppedHook"}, "match_mode":
"regex", "controls": {"hook_lineage": false}},
+ {"scope": {"operator": ".*\\.ExcludeDatasetsConfOperator"}, "match_mode":
"regex", "controls": {"exclude_datasets": [".*/operator_excluded_.*"]}},
+ {"scope": {"operator": ".*\\.ExcludeDatasetsConfOperator", "hook":
".*\\.ConfHook"}, "match_mode": "regex", "controls": {"exclude_datasets":
[".*/operator_hook_excluded_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_conf_dag",
"hook": "airflow.providers.standard.hooks.filesystem.FSHook"}, "controls":
{"exclude_datasets": [".*/fs_hook_excluded_.*"]}},
+ {"scope": {"hook": ".*\\.ExcludeDatasetsGlobalHook"}, "match_mode":
"regex", "controls": {"exclude_datasets": [".*/hook_excluded_.*"]}},
+ {"scope": {"hook": ".*\\.ExcludeDatasetsGlobalOtherHook"}, "match_mode":
"regex", "controls": {"exclude_datasets": [".*/other_hook_excluded_.*"]}},
+ {"scope": {"operator": ".*\\.ExcludeDatasetsGlobalOperator"},
"match_mode": "regex", "controls": {"exclude_datasets":
[".*/operator_excluded_.*"]}},
+ {"scope": {"operator": ".*\\.ExcludeDatasetsGlobalOperator", "hook":
".*\\.ExcludeDatasetsGlobalHook"}, "match_mode": "regex", "controls":
{"exclude_datasets": [".*/operator_hook_excluded_.*"]}},
+ {"scope": {"operator": ".*\\.ExcludeDatasetsGlobalClearOperator"},
"match_mode": "regex", "controls": {"exclude_datasets": []}},
+ {"scope": {"hook": ".*\\.ExcludeDatasetsGlobalDropHook"}, "match_mode":
"regex", "controls": {"hook_lineage": false}},
+ {"scope": {"operator": ".*\\.ExcludeDatasetsGlobalOperator", "hook":
".*\\.ExcludeDatasetsGlobalOpDropHook"}, "match_mode": "regex", "controls":
{"hook_lineage": false}},
+ {"scope": {"operator": ".*\\.ExcludeDatasetsGlobalLockedOperator"},
"match_mode": "regex", "locked": true, "controls": {"exclude_datasets":
[".*/operator_locked_.*"]}},
+ {"scope": {"hook": ".*\\.ExcludeDatasetsGlobalLockedHook"}, "match_mode":
"regex", "locked": true, "controls": {"exclude_datasets":
[".*/hook_locked_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_sources_dag"},
"controls": {"exclude_datasets": [".*/dag_excluded_.*", ".*sql_excluded.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_sources_dag",
"hook": ".*\\.SqlRuledHook"}, "match_mode": "regex", "controls":
{"exclude_datasets": [".*sql_hook_excluded.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_sources_dag",
"hook": ".*\\.SqlDroppedHook"}, "match_mode": "regex", "controls":
{"hook_lineage": false}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_sources_dag",
"hook": ".*\\.SourceHook"}, "match_mode": "regex", "controls":
{"exclude_datasets": [".*/hook_scoped_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_versioned_dag"},
"controls": {"exclude_datasets": [".*/dag_excluded_.*"]}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_versioned_dag",
"task_id": "t_skipped_emit_off"}, "controls": {"emit": false}},
+ {"scope": {"dag_id": "openlineage_policy_exclude_datasets_versioned_dag",
"task_id": "t_skipped_full_info"}, "controls": {"include_full_task_info": true}}
+ ]
AIRFLOW__OPENLINEAGE__EXECUTION_TIMEOUT: '60'
+ # openlineage_policy_exclude_datasets_versioned_dag checks that exclusion
runs after this cap; no other DAG
Review Comment:
Done, I used your wording.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
