Hi Airflowers:
I was testing using the S3KeySensor and it became clear that it's using
boto instead of boto3. Is there any plan to migrate to boto3 (2.0 for
instance especially if there is full support for python 3)?
https://aws.amazon.com/blogs/aws/now-available-aws-sdk-for-python-3-boto3/
Than
thur for the status.
>
> Cheers,
> Chris
>
> On Wed, May 18, 2016 at 3:09 PM, David Klosowski
> wrote:
>
> > Hi Airflowers:
> >
> > I was testing using the S3KeySensor and it became clear that it's using
> > boto instead of boto3. Is there any plan
I have a DAG that I just deployed that the scheduler keeps scheduling for
the last two months in the past.
start_date: 8/5/2016
scheduled runs started:
7/3/2016
6/5/2016
Here is the gist of this DAG's architecture:
The DAG depends another dags tasks using 7 dynamic ExternalTaskSensors that
it b
does not get updated. I work
> around this by changing the name of my dag. I do lose history as well, but
> it works.
>
> My dags are named "some_dag_v1". When I change a start date, I update the
> version suffix to force a reload : "some_dag_v2"
>
> -s
>
&
clear to me what you want to do exactly.
> -s
>
> On Tue, Aug 2, 2016 at 8:02 PM, David Klosowski
> wrote:
>
> > start_date being updated isn't the issue here. I haven't changed it.
> New
> > execution_dates keep getting created for the past before any da
Running 1.7.1.3 currently. There are certainly a large number of commits.
What's the current period of releasing new versions? It might be good to
release new versions more frequently.
Cheers,
David
On Wed, Aug 3, 2016 at 12:37 PM, Sean Cronin wrote:
> Hey everyone,
>
> What's needed to cut a
I've been trying to see if there is a way to setup airflow to use github
outh for authentication with Github.
https://developer.github.com/v3/oauth/
I've read through:
http://airflow.incubator.apache.org/security.html
And I see a GitHub Enterprise Authentication section, so I tried to set
this
We're running airflow using docker + ecs with EMR and Redshift and have not
noticed any issues with the PostgresOperator yet. We don't have anything
running in production with Redshift, but have evaluated a few DAGs using
it. I haven't noticed any issued with the Operator. I would also look at
t
Hi All,
In 1.7.1.3 I had experienced the exact issue described here:
https://issues.apache.org/jira/browse/AIRFLOW-736
The underlying task instances of a SubDag are complete; however, the SubDag
itself shows as running and is not marked as successful (until manually
done so). It was an intermit
I was curious what the thoughts are regarding shifting Airflow to a publish
model from the poll model currently used by many of the sensors inheriting
from BaseSensorOperator.
It would prevent much of the polling that may take up quite a bit of
resources.
Thanks.
Regards,
David
Hi Airflow Dev List:
Has anyone had cases where tasks get "stuck"? What I mean by "stuck" is
that tasks show as running through the Airflow UI but never actually run
(and dependent tasks will eventually timeout).
This only happens during our deployments and we replace all the hosts in
our stack
Another way is to update the database inbetween, but I'd personally avoid
> that as much as you can.
>
>
> Not sure if anyone wants to chime in here on how to best deal with this in
> docker?
>
> Rgds,
>
> Gerard
>
>
> On Mon, Aug 7, 2017 at 8:21 PM, David K
Hey Airflow Devs:
We noticed an issue with task instances not running sometimes.
Some interesting details are:
> in the database it shows that specific tasks that are "stuck" have an
end_date but no state (null state)
> Airflow has been running for several days just fine and there were no
changes
t; > On 22 Sep 2017, at 19:54, David Klosowski wrote:
> >
> > Hey Airflow Devs:
> >
> > We noticed an issue with task instances not running sometimes.
> >
> > Some interesting details are:
> >> in the database it shows that specific tasks that are "
Hi Airflow Devs:
Is there any chance that we can include AIRFLOW-1397 and AIRFLOW-1483 as
part of the 1.9.0 release?
We've noticed some UI changes after 1.8.0 and it affects our ability to
deal with the status and errors of DAGs. The UI has become quite important
to us, so these changes were a b
Great, thanks Chris.
Cheers,
David
On Fri, Sep 29, 2017 at 9:26 AM, Chris Riccomini
wrote:
> They are both already in 1.9.0.
>
> On Thu, Sep 28, 2017 at 6:15 PM, David Klosowski
> wrote:
>
> > Hi Airflow Devs:
> >
> > Is there any chance that we can include
Hi Edgar,
Thanks. I think that will be quite helpful. We're likely to start running
the v1-9-test branch in production if it has all the fixes we're hoping for
(going to evaluate the PRs).
Upgrading from 1.8.0 -> 1.8.2 hasn't been as smooth as we would have
liked. A good number of things have
Hey Airflow Devs:
How is s3 logging supposed to work in Airflow 1.9.0?
I've followed the *UPDATING.md* guide for the new setup of logging and
while I can use my custom logging configuration module to format the files
written to the host, the s3 logging doesn't appear to work as I don't see
anythi
Not sure if this is the issue, since I'm using the CeleryExecutor
https://issues.apache.org/jira/browse/AIRFLOW-1667
Interestingly enough it doesn't work with the LocalExecutor either.
Regards,
David
On Fri, Oct 6, 2017 at 4:11 PM, David Klosowski
wrote:
> Hey Airflow Devs:
ss=mymodule.LOGGING_CONFIG
> ```
>
>
>
>
> > On 7 Oct 2017, at 00:11, David Klosowski wrote:
> >
> > Hey Airflow Devs:
> >
> > How is s3 logging supposed to work in Airflow 1.9.0?
> >
> > I've followed the *UPDATING.md* guide for the new setup
ork as you can't share the task logs at the host level
when the containers are distributed.
Thanks.
Regards,
David
On Sat, Oct 7, 2017 at 10:06 AM, David Klosowski
wrote:
> Hi Ash,
>
> Thanks for the response .
>
> I neglected to post that I do in fact have that config
> On Mon, Oct 9, 2017 at 10:39 AM, Chris Riccomini
> wrote:
>
> > Have a look at this:
> >
> > https://github.com/apache/incubator-airflow/pull/2671
> >
> > I had to do a similar dance.
> >
> >
> > On Mon, Oct 9, 2017 at 10:28 AM, David
Hey Airflow Devs,
We've noticed that the on_retry_callback on the SubDagOperator seems to be
acting differently in version 1.9.0 from 1.8.x. What's more interesting,
it's only happening in one environment and not another making it hard to
determine the issue since that environment that has unexpe
Sorry, I meant to say we check the state of FAILED and not UPSTREAM_FAILED
tasks, which we'd set to UP_FOR_RETRY.
Thanks.
Regards,
David
On Mon, Oct 23, 2017 at 10:30 AM, David Klosowski
wrote:
> Hey Airflow Devs,
>
> We've noticed that the on_retry_callback on the SubDagO
Hey Airflow Devs:
The following commit for, AIRFLOW-1641, recently added breaks the scheduler
in Airflow
Master:
https://github.com/apache/incubator-airflow/commit/2abead7049806482047e29d123a109b444c00355
v1-9-test:
https://github.com/apache/incubator-airflow/commit/73549763eac74142b7c4018422bb2
I've created this bug via Jira:
https://issues.apache.org/jira/browse/AIRFLOW-1767
Thanks.
Regards,
David
On Tue, Oct 31, 2017 at 9:45 AM, David Klosowski
wrote:
> Hey Airflow Devs:
>
> The following commit for, AIRFLOW-1641, recently added breaks the
> scheduler in A
I noticed that the PR builds in TravisCI are failing with the following
apt-get error:
--2018-03-30 17:56:23-- (try: 5)
http://download.oracle.com/otn-pub/java/jdk/8u161-b12/2f38c3b165be4555a1fa6e98c45e0808/jdk-8u161-linux-x64.tar.gz
Connecting to download.oracle.com
(download.oracle.com)|23.45.
--:--
> 1754
>
> 100 180M 100 180M0 0 12.7M 0 0:00:14 0:00:14 --:--:--
> 13.5M
>
> - Jin
>
> On Fri, Mar 30, 2018 at 11:27 AM, David Klosowski
> wrote:
>
> > I noticed that the PR builds in TravisCI are failing with the following
> &
Bumping this since there has yet to be a resolution here.
On Fri, Mar 30, 2018 at 3:43 PM, David Klosowski
wrote:
> Created this ticket to track this:
>
> https://issues.apache.org/jira/browse/AIRFLOW-2272
>
> On Fri, Mar 30, 2018 at 3:15 PM, Jin Chang wrote:
>
>> It
Hi Airflow Devs:
Is this timezone issue in Airflow version 1.10.0 or only in 1.10.1?
Thanks.
Regards,
David
On Tue, Oct 30, 2018 at 11:11 AM Bolke de Bruin wrote:
> we specifically remove timezone info to determine the next schedule. Ie.
> cron sets exact date times so tz info should not make
30 matches
Mail list logo
