maxcotec commented on issue #56130:
URL: https://github.com/apache/airflow/issues/56130#issuecomment-3353884621
faced same issue today, and this is the error i was getting on dag-processor;
`│ sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1038, 'Out
of sort memory, consider increasing server sort buffer size') │ │ [SQL: SELECT
serialized_dag.data, serialized_dag.data_compressed, serialized_dag.id,
serialized_dag.dag_id, serialized_dag.created_at, serialized_dag │ │ FROM
serialized_dag │ │ WHERE serialized_dag.dag_id = %s ORDER BY
serialized_dag.created_at DESC │ │ LIMIT %s] │ │ [parameters:
('relational_v2_billing_live_db', 1)] │ │ (Background on this error at:
https://sqlalche.me/e/14/e3q8)`
what i found was no index for `WHERE dag_id = ? ORDER BY created_at DESC
LIMIT 1 on serialized_dag`, whic is probably hitting/returning larger
serialized rows more often.
Just added right index so MySQL can fetch the newest row without sorting:
```
ALTER TABLE serialized_dag
ADD INDEX idx_sd_dag_created (dag_id, created_at);
ANALYZE TABLE serialized_dag;
```
check;
```
EXPLAIN SELECT id
FROM serialized_dag
WHERE dag_id='Hello_dag'
ORDER BY created_at DESC
LIMIT 1;
```
here now you should expect key=idx_sd_dag_created, Extra: Backward index
scan; Using index.
After this, my dag-processor never complaint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
