Apologies for the self-reply - however, was digging in a bit more to the visibility timeout - if that is indeed the case, then I had some more info/questions:
- Is there a way to patch this on our own build via the following on 1.8.1 - I see the following pull request/commits as a way to pass in BROKER_TRANSPORT_OPTIONS to set visibility_timeout: https://github.com/apache/incubator-airflow/pull/2143 <https://github.com/apache/incubator-airflow/pull/2143> which points to the following PR: https://issues.apache.org/jira/browse/AIRFLOW-966 <https://issues.apache.org/jira/browse/AIRFLOW-966> - Is there a way to get a fix for this issue (with the above patches or otherwise if it is a trivial patch to allow for passing visibility_timeout to the celery workers/redis) into 1.8.2 before release?! :) Cheers, J > On Jul 16, 2017, at 10:36 AM, Jawahar Panchal <[email protected]> wrote: > > Awesome - thank you for the help. Does this have anything to do with the > default 1h visibility timeout when using redis as a broker? Have been doing > some more digging, and was not aware of this caveat - however, am not sure if > this behavior is what leads to the symptoms I am seeing (as well as what is > documented in the bug report)… > > Thanks again for the help - much appreciated! > > Cheers, > J > >> On Jul 16, 2017, at 3:22 AM, Alex Guziel <[email protected]> >> wrote: >> >> I think this may be related to a celery bug. I'll follow up with more >> details later. >> >> On Sun, Jul 16, 2017 at 12:56 AM Jawahar Panchal <[email protected]> >> wrote: >> >>> Hi! >>> >>> I am currently running a couple of long-running tasks on a >>> database/dataset at school for a project that results in behavior/log >>> output similar to what was flagged in this bug: >>> https://issues.apache.org/jira/browse/AIRFLOW-1258 < >>> https://issues.apache.org/jira/browse/AIRFLOW-1258> >>> >>> Wasn’t sure if anyone on the list had seen anything similar, or would know >>> what I can do to possibly debug further/patch. As it takes 1hr to test a >>> change, needless to say any pointers from the dev team on the right >>> direction to look within the codebase would be much appreciated! :) >>> >>> Thanks in advance for everyone’s/anyone's time and help - am not an >>> Airflow expert, but am hopefully learning quickly enough to help resolve >>> this issue (if I am ‘barking up the right tree’ with this bug number…) >>> >>> Cheers, >>> J >>> >>> >
