pierrejeambrun commented on code in PR #58814:
URL: https://github.com/apache/airflow/pull/58814#discussion_r2580550501
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -712,6 +712,20 @@ database:
type: integer
example: ~
default: "10000"
+ metadata_indexes:
+ description: |
+ JSON list of additional indexes to create on the metadata database at
API server startup.
+
+ Each item must be a string specifying the table and one or more
columns:
+ - "table(column1, column2, ...)"
+
+ Existing indexes are detected and skipped. On PostgreSQL, indexes are
created
+ CONCURRENTLY to avoid locking tables. Other databases attempt
non-blocking creation
+ where supported, otherwise fallback to standard index creation.
+ version_added: 3.2.0
+ type: string
+ example: "task_instance(dag_id, task_id,
run_id)|log(dttm)|dag_run(dag_id, run_id)"
Review Comment:
> I'd suggest we instead build a way to provision these indexes we've
identified at component startup.
What do you mean by that? To give some context the idea was that we do not
know how people will use the API (specific filtering/ordering) UI (specific
list views), and we cannot create all the possible indexes / columns
combination indexes. Also it depends on their tables size some people will have
a really huge DagRun table and need some indexes there, some people will have
that problem on the dag table etc... So the idea was to let them decide what
they need depending on their usage and provide a way to help them create the
index. Can you give more details to what you have in mind here?
> if someone knows what indexes to add, they can just add them... This feels
like a footgun.
That is true, I thought about that too at some point.
Maybe just expending the doc with common use cases of why people would want
to add specific indexes to the db and how to identify those bottlenecks is
enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
