dimon222 edited a comment on issue #17897: URL: https://github.com/apache/airflow/issues/17897#issuecomment-910851491
Here's what I tried just now (one at a time, of course) 1. Above mentioned manager.py correction 2. Restrict amount of parsing processes to 1. 3. Completely bring scheduler down for time of updating dag tags, then bring it back up 4. Increase min_file_process_interval 5. Change "use_row_level_locking" to False (I have it as True initially) 6. Change sort mode to alphabetical 7. Tune other timeouts In all the possible options I'm receiving the above mentioned SQL error. If I decrease amount of processes to 1 the scheduler immediately chonks, spits this error and exits with non-zero exit code. I have also enabled debug logging and tried to see what DAGs are being picked, and I don't seem to notice anything that might reflect concurrency problems (or picking up same dag twice). Unless, the problem is with logic that does bulk insert ? Like doubling amount of queries or such. I tried for disable Sqlalchemy pool, print all the tags it detects - everything seems fine from collected logs, but issue still arises. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
