Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC3

pip can be very picky at times the version number of things it installs, specifically when installing from wheels (which are pre-packaged, and recommend because they are much faster to install.) A distribution version number should follow PEP-0440 https://www.python.org/dev/peps/pep-0440/

Re: [webserver] Webserver time_starttransfer very high

(Your screenshot didn't come through for me, possibly because the list stripped it? That said:) Is it always 6 seconds, or after making a few requests, enough so that each worker stands a chance to have loaded the app any deps does it settle down? i.e. the problem might just be that of

Re: Proposal: Set Celery 4.0 as a minimum as Celery 4 is unsupported

+1 from us, we're running on Celery 4.0.2 in production on Airflow 1.8.2 (4.1 wasn't out when we started and haven't upgraded in prod yet) > On 16 Sep 2017, at 16:35, Bolke de Bruin wrote: > > Hi, > > Some refactoring of the Celery config is underway and as some of the

Re: 1.9.0 test branch has been cut

There are 14 (only 2 of which are open) issues that are marked with a fix version of 1.8.3. What should happen to those? > On 13 Sep 2017, at 20:45, Chris Riccomini wrote: > > Hey all, > > I've cut a 1.9.0 test branch. > >

Re: 1.9.0 test branch has been cut

ll (we can have multiple fix versions). > > On Wed, Sep 13, 2017 at 12:52 PM, Ash Berlin-Taylor < > ash_airflowl...@firemirror.com> wrote: > >> There are 14 (only 2 of which are open) issues that are marked with a fix >> version of 1.8.3. What should happen to those

Re: 1.9.0 PR mini-rundown

gt;>>>>>>> Hey all, >>>>>>>> >>>>>>>> The workflow is documented here: >>>>>>>> >>>>>>>> >>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Releasing+Airflow >

Re: Terminate task process through UI

That sets the task instance state to SHUTDOWN, which is then picked up by the task sentinel here https://github.com/apache/incubator-airflow/blob/b2e1753f5b74ad1b6e0889f7b784ce69623c95ce/airflow/jobs.py#L164-L165

Re: Upgrading to 1.8.2 fails to display variable page

Was it included as an image? If so it was stripped (and would be more useful included as text anyway) -ash > On 7 Sep 2017, at 17:12, David Capwell wrote: > > I just upgraded a test environment from 1.8.0 to 1.8.2 and notice that the > variables page is no longer able to

Re: Using s3 logging in Airflow 1.9.x

It could be that you have created a custom logging file, but you haven't specified it in your airflow.cfg: ``` logging_config_class=mymodule.LOGGING_CONFIG ``` > On 7 Oct 2017, at 00:11, David Klosowski wrote: > > Hey Airflow Devs: > > How is s3 logging supposed to

Re: [VOTE] Airflow 1.9.0rc1

The other difference is that if you depend upon airflow in a module where you want to put it in the install_requires section of a setup.py (not an application which has a requirements.txt, say) you can't use a git tag. Or at least I couldn't get it working. It doesn't make a difference a lot

Re: [VOTE] Airflow 1.9.0rc4

n <bdbr...@gmail.com> wrote: > > airflow.task.raw points to airflow.task by default, which does have a handler > and that is the one that is travelled. And yes we know the code is incorrect > in this area, but the default does work correctly. > > Bolke. > > >>

Re: [VOTE] Airflow 1.9.0rc4

>>> Op 30 nov. 2017 om 18:17 heeft Chris Riccomini <criccom...@apache.org> >> het volgende geschreven: >>> >>> @bolke/@fokko, thoughts? >>> >>> As an aside, we had an issue seeing task logs, but were able to fix it by >>> modifying our logging config

Re: [VOTE] Airflow 1.9.0rc4

me. -ash > On 30 Nov 2017, at 17:03, Chris Riccomini <criccom...@apache.org> wrote: > > @Ash, do you think this issue is blocking for 1.9.0? > > On Thu, Nov 30, 2017 at 8:22 AM, Ash Berlin-Taylor < > ash_airflowl...@firemirror.com> wrote: > >> Ah,

Re: [VOTE] Airflow 1.9.0rc4

add an explicit handler for .task.raw in my logging config. - We can always call set_context on airflow.task instead of airflow.task.raw - We walk up the .parent links if .propagate is True (possibly stopping once we find any handlers) -ash > On 30 Nov 2017, at 16:02, Ash Berlin-Tay

Re: [VOTE] Airflow 1.9.0rc4

We're running more tests (day of meetings meant we couldn't do as many as I'd have liked yesterday) but looking good so far. +0.5 (non binding) -ash > On 29 Nov 2017, at 18:31, Chris Riccomini wrote: > > Anyone else? Last day before I close the vote off. > > On Mon,

Re: [VOTE] Airflow 1.9.0rc4

er fix to do that instead. -ash > On 30 Nov 2017, at 23:18, Ash Berlin-Taylor <a...@firemirror.com> wrote: > > Thanks for picking that up so speedily Bolke! > > I've opened a jira ticket for the other problem I found: > https://issues.apache.org/jira/browse/AIRFLOW-1873

Re: [VOTE] Airflow 1.9.0rc4

as to >>> what's going on. >>> >>> @bolke/@ash, thanks for looking into this! >>> >>> On Thu, Nov 30, 2017 at 10:23 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: >>> >>>> Ok, I think I can confirm the issue. I don’t see an

Re: [VOTE] Airflow 1.9.0rc4

(), or `ti.log.info()` Anyone else noticed similar, or is it something specific to our environment? -ash > On 30 Nov 2017, at 09:52, Ash Berlin-Taylor <ash_airflowl...@firemirror.com> > wrote: > > We're running more tests (day of meetings meant we couldn't do as many as I'd >

Re: [VOTE] Airflow 1.9.0rc6

;> https://github.com/apache/incubator-airflow/pull/2668/files but I do >> not >>>> personally consider this a blocker for rc6 release. >>>> >>>> >>>>> On Wed, Dec 13, 2017 at 8:49 AM Feng Lu <fen...@google.com.invalid> >>> wro

Backwards compability - what do we mean? when? how long?

Hi, A question came up on a github issue about what exactly we meant about backwards compatibility, and I figured we as a project should work out what we mean when we say we want to maintain compat. And most importantly document it (don't worry, I'm volunteering to do bit, so long as we reach

Re: Making Airflow Timezone aware

This sounds like an awesome change! I'm happy to review (will take a look tomorrow) but won't be a suitable tester as all our DAGs operate in UTC. -ash > On 13 Nov 2017, at 18:09, Bolke de Bruin wrote: > > Hi All, > > I just want to make you aware that I am creating

Re: [VOTE] Airflow 1.9.0rc3

-0.5, from me again (Sorry!) We got as far as deploying this to our test cluster and ran into another bug in the S3Hook: `list_keys` doesn't work (and I suspect list_prefixes will be similarly broken, though we don't use it ourselves): File

Re: [VOTE] Airflow 1.9.0rc2

https://issues.apache.org/jira/browse/AIRFLOW-1795 is still open which breaks a number of operators from Airflow 1.8.2 to 1.9.0: RedshiftToS3 S3FileTransformOperator S3KeySensor S3PrefixSensor S3ToHive I present a couple of ways of solving

Re: Docs

Not that it helps fix the "official" site, but http://airflow.readthedocs.io/en/latest/installation.html contains more up-to-date docs. (Though I'm not sure what versions this doc relates to: the "versions" selection in the bottom

Re: Airflow configuration in environment variable not working

Connections are configured differently: http://airflow.readthedocs.io/en/latest/concepts.html#connections > When referencing the connection in the Airflow pipeline, the conn_id should > be the name of the variable without the prefix. For example, if the `conn_id` > is named `postgres_master`

Re: [VOTE] Airflow 1.9.0rc1

gt;> running SSH properly), some minor fixes by Airbnb should also be in RC2. >> There is some logging “weirdness”, that might warrant a small patch here in >> there and could be squeezed into RC2, but I don’t consider them blocking. >> >> So almost there, but we need an

Re: [VOTE] Airflow 1.9.0rc1

tests for S3Hook which it looks like was kind of lacking. -ash > On 9 Nov 2017, at 10:54, Ash Berlin-Taylor <ash_airflowl...@firemirror.com> > wrote: > > Thanks for picking this up. Your fix should stop the 500 error, but there's > another problem (which is ultimately

Re: [VOTE] Airflow 1.9.0rc1

I'd be in favour of this for similar reasons to Alek. I think the "mentioned before" is in reference to my post from October 20 http://mail-archives.apache.org/mod_mbox/incubator-airflow-dev/201710.mbox/%3CD008C556-C67C-42EB-88C0-CFE440C93656%40firemirror.com%3E

Re: [VOTE] Airflow 1.9.0rc1

dlers to try loading under the old pattern if none are found with the new style? (The other option is that I just run a migration script to move the old logs into the new place. That sort of only helps me though.) -ash > On 9 Nov 2017, at 11:04, Ash Berlin-Taylor <ash_airflowl...@firemi

Re: [VOTE] Airflow 1.9.0rc1

-1 (for now. Non binding. Is that how this process works?) We've built a test env for this RC and are testing, but have run into an issue reading task logs. (See below) We haven't gotten very far with this yet, we will dig more tomorrow (it's the end of the UK work day now). I suspect this

Re: Xcom_push is broken - probably after the timezone awareness

Serialising XCom via pickle is being deprecated for Airflow 2 btw: https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;a=commitdiff;h=4cf904c;hp=984a87c0cb685ea4dfa765cc4f4a23c9058b3965

Re: [VOTE] Airflow 1.9.0rc6

+0.5 from me. Our big test will come on Thursday morning, but looking good so far for the small daily dags we've got are running okay, logs are showing up, and making their way to S3. -ash > On 11 Dec 2017, at 18:50, Chris Riccomini wrote: > > Hey all, > > I have cut

Re: Experimental API

Oh gods. Something has gone wrong - the methods are decorated with `@requires_authentication` but they... don't. Oh, because the default backend doesn't do any authentication or protection at all. I thik this is CVEworthy - using the User+Password auth for the web front end/using default

Re: Experimental API

gt; On Mon, Oct 30, 2017 at 11:42 AM, Ash Berlin-Taylor < > ash_airflowl...@firemirror.com> wrote: > >> Oh gods. >> >> Something has gone wrong - the methods are decorated with >> `@requires_authentication` but they... don't. Oh, because the default >

Re: Experimental API

t; > https://github.com/apache/incubator-airflow/pull/2734 > > Niels > > Op 30 okt. 2017 4:58 p.m. schreef "Ash Berlin-Taylor" < > ash_airflowl...@firemirror.com>: > >> It's available by default. >> >> https://github.com/apache/incubator-ai

Re: ImportError: No module named sendgrid

1. PRs generally aren't "approved" in Github if a core contributor looks at it and is happy 2. It was merged by criccomini, not the opener https://github.com/apache/incubator-airflow/commit/7cb818bbacb2a2695282471591a9e323d8efbf5c 3, 4, 5. I agree, and I made the same point

Re: ImportError: No module named sendgrid

1. PRs generally aren't "approved" in Github if a core contributor looks at it and is happy 2. It was merged by criccomini, not the opener https://github.com/apache/incubator-airflow/commit/7cb818bbacb2a2695282471591a9e323d8efbf5c

Re: Airflow 1.9.0 status

I think we need to add https://issues.apache.org/jira/browse/AIRFLOW-1764 https://issues.apache.org/jira/browse/AIRFLOW-1765 to that list as critical security fixes. I'll tackled 1765 tomorrow (UK time) if no one gets round to it before then. -ash > On 30 Oct 2017, at 17:58, Chris Riccomini

Re: How to bes use Google Cloud Storage for logging?

What version are you on? I can't match up the line numbers in this stack trace to either 1.9.0rc8 or 1.9.0rc2 -- both of which show the 'if old_log else log' on line 157 -ash > On 20 Dec 2017, at 15:25, Kevin Lam wrote: > > Thanks Bolke and Feng! > > I seem to have a

Re: Apache Airflow welcome new committer/PMC member : Naik Kaxil (a.k.a. kaxil)

Welcome! On the github side: none of the commiters (currently) have permission to merge/edit tickets on Githb. Recently-ish Apache made it possible for projects to use Github as the primary repo, and I called a vote before I stopped for paternity leave (which I'm coming to the end of now). The

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

subject of Pr what the associated issues was, since we moved >> history became a lot cleaner and changelogs are now easy to generate. >> >> My 2cents >> >> B. >> >> Verstuurd vanaf mijn iPad >> >>> Op 9 mrt. 2018 om 18:01 heeft Ash Berlin

Re: celery problem: cannot override celery_broker_transport_options

oker_transport_options] are commented out? > And if someone is running Redis, they would have to add those > options in their own airflow.cfg file? > > Bolke, do you have any comments? > > -- > Craig > > On Tue, May 22, 2018 at 1:50 AM Ash Berlin-Taylor < > ash_air

Re: celery problem: cannot override celery_broker_transport_options

s) >> 'ssl_key','ssl_cert','ssl_active','visibility_timeout','ssl_cacert' sent to >> create_engine(), using configuration >> MySQLDialect_mysqldb/QueuePool/Engine. Please check that the keyword >> arguments are appropriate for this combination of components. >

Re: celery problem: cannot override celery_broker_transport_options

7909a652e6ded2 >> >> I cannot find the pull request or release notes which document this. >> >> -- >> Craig >> >> >> On Thu, May 24, 2018 at 8:19 AM Ash Berlin-Taylor < >> ash_airflowl...@firemirror.com> wrote: >> >>> Sounds li

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

subject of Pr what the associated issues was, since we moved >> history became a lot cleaner and changelogs are now easy to generate. >> >> My 2cents >> >> B. >> >> Verstuurd vanaf mijn iPad >> >>> Op 9 mrt. 2018 om 18:01 heeft Ash Berlin

Re: celery problem: cannot override celery_broker_transport_options

To use with the SQLA backend to celery you need to override the options Airflow passes to Celery. Those come from https://github.com/apache/incubator-airflow/blob/v1-10-test/airflow/config_templates/default_celery.py Since you don't want most/all of those options (and there is no way in the

Re: Is `airflow backfill` disfunctional?

Somewhat related to this, but likely a different issue: I've just had a case where a long (7hours) running backfill task ended up running twice somehow. We're using Celery so this might be related to some sort of Celery visibility timeout, but I haven't had a chance to be able to dig in to it

Re: [VOTE] [RESULT] Migrate to Github as primary repo (a.k.a. Gitbox)

t, thanks for doing this Ash! > > On Wed, Mar 21, 2018 at 8:14 AM, Ash Berlin-Taylor < > ash_airflowl...@firemirror.com> wrote: > >> The vote passed with the following +1's, no -1. >> >> 4+ Binding (PPMC) votes >> Ash Berlin-Taylor, Chris Riccomin

Re: conn_id breaking change; once more with feeling

I was involved in the Github discussion about the rename to aws_conn_id, and it prompted me to write http://mail-archives.apache.org/mod_mbox/airflow-dev/201801.mbox/%3cCABYbY7dPS8X6Z4mgbahevQwF5BnYYHXezFo=avoLBNxPzp5=b...@mail.gmail.com%3e

Re: Airflow London Meetup Group

Hey, I'd be interested in this, and I'm sure I could drag a few colleagues along too :) -ash > On 31 May 2018, at 20:55, Naik Kaxil wrote: > > Hi all, > > We(me and my colleagues at Data Reply UK) are planning to start a meetup for > Airflow users. Me & one of my friend already gave a

Re: Log external module into airflow log

From inside a running task these should already show up in the task instance log. Do they not? Or do you want the logging from parsing the DAG itself? These probably get routed to $AIRFLOW_HOME/logs/scheduler/latest/$dag_id.log (path approximate - from memory) -ash > On 1 Jun 2018, at 19:02,

Re: Apache Airflow 1.10.0b2

pat with 1.9 config names. > On 26 Jun 2018, at 09:05, Ash Berlin-Taylor > wrote: > > I'm preparing a patch that maintains config compat between 1.9 and 1.10 > around Celery settings. We renamed them to better names in AIRFLOW-1840 but > we should respect the old names fo

Re: Apache Airflow 1.10.0b2

I'm preparing a patch that maintains config compat between 1.9 and 1.10 around Celery settings. We renamed them to better names in AIRFLOW-1840 but we should respect the old names for at least one release version. -ash > On 25 Jun 2018, at 22:30, Stefan Seelmann wrote: > > I noticed one bug,

Publishing alpha releases to PyPI?

(Bit of a long mail, skip to the end for specific ask, and why I think it is safe) Yesterday we tried upgrading a test cluster of ours to the alpha of 1.9.0 but ran into a few snags around installation. We don't actually install Airflow directly; instead we install our in-house python module

Two small DAG chart ui fixes

I just opened two small PRs to fix some minor (but annoying) glitches on the DAG task duration page: Task Duration shows two charts on first page load https://issues.apache.org/jira/browse/AIRFLOW-1741 NVD3 Charts do not have labeled axes and units change dynamically

Re: Fix on_kill command for operators

Without this change does on_kill ever get triggered? It seems like this change is desired behaviour. As per the first comment https://issues.apache.org/jira/browse/AIRFLOW-1623?focusedCommentId=16171819=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16171819

Re: Installation of Airflow

If you want to use system installed modules (such as os-packaged py libs) the other thing you can do is `pip install --user apache-airflow` which will install into ~/.local/bin and ~/.local/lib etc. This is automatically in the python search path ahead of the system paths for the user (This is

Re: Double Logging to GCS? (And to S3)

Nope, probably not :( -ash > On 13 Jan 2018, at 18:51, Bolke de Bruin <bdbr...@gmail.com> wrote: > > We shouldn’t have missed that :-( > >> On 13 Jan 2018, at 19:49, Ash Berlin-Taylor <ash_airflowl...@firemirror.com> >> wrote: >> >> This is a

Re: Double Logging to GCS? (And to S3)

This is a known bug in 1.9.0 that we didn't manage to fix before release -- the fix for it was https://github.com/apache/incubator-airflow/commit/4ce4faaeae7a76d97defcf9a9d3304ac9d78b9bd (merged in

Re: max_active_runs

It seems unlikely, but could it be the location of where max_active_runs is specified? In our DAGs we pass it directly as an argument to the DAG() call, not via default_arguments and it behaves itself for us. I think I should check that! -ash > On 14 Feb 2018, at 13:43, Gerard Toonstra

Re: Airflow Documentation - Readthedocs

Relatedly: can we remove the docs for the old "airflow" dist on pypi? If someone has permissions to manage the on pypi, can you go to https://pypi.python.org/pypi?%3Aaction=pkg_edit=airflow and hit the "Destroy Documentation" button: >

Re: Scheduler won't schedule past minimum end_date of tasks

That does sound like a bug, and I would have expected, as you did, that not specifying an end_date on some tasks means those tasks should run for ever. Changes that probably need making is that a task end_date of None on a task should me "greater" than other task dates in/around the lines you

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

fra" ¯\_(ツ)_/¯ -ash > On 24 May 2018, at 16:30, Ash Berlin-Taylor > wrote: > > Found some time, opened a PR: > https://github.com/apache/incubator-airflow/pull/3413 > <https://github.com/apache/incubator-airflow/pull/3413> > > Commiters: > > -

Re: Tensorflow Hub - cant seem to load module in Airflow

> > https://github.com/tensorflow/hub/issues/76 > > Seems like maybe Airflow is passing as bytes when it's being expected as a > string..? > > On Tue, Jun 19, 2018 at 11:02 AM Ash Berlin-Taylor < > ash_airflowl...@firemirror.com> wrote: > >> Welp, nothing u

Re: Tensorflow Hub - cant seem to load module in Airflow

urlunparse > [2018-06-19 09:45:28,368] {base_task_runner.py:98} INFO - Subtask: > _coerce_args(*components)) > [2018-06-19 09:45:28,370] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/future/backports/urllib/parse.py", > line 115, i

We've migrated to Github to repo!

Hi everyone, but especially committers: we have now moved to Github, and should be able to commit directly there (and close issues too hopefully). If you are a committer and haven't yet linked your Github account with ASF go to https://gitbox.apache.org/setup/ and follow the instructions. If

Re: We've migrated to Github to repo!

It appears we also have comments on Github issues being auto-duplicated to the dev mailing list -- this will increase the email volume on that list. Would we like to keep that feature or disable it? -ash > On 30 Jul 2018, at 20:24, Ash Berlin-Taylor > wrote: > > Hi everyone, bu

Re: We've migrated to Github to repo!

Hi Sergi, Yes, I agree, and I've asked (as a short term measure) for the comments to list and comments on every PR duplicating to Jira to be removed (so that the volume was as it was) -- we can then think about what we want to re-enable, perhaps to a different list. Fokko: the new

Re: We've migrated to Github to repo!

> On 1 Aug 2018, at 09:43, Ash Berlin-Taylor > wrote: > > Hi Sergi, > > Yes, I agree, and I've asked (as a short term measure) for the comments to > list and comments on every PR duplicating to Jira to be removed (so that the > volume was as it was) -- we can then

Re: The need for LocalTaskJob

Comments inline. > On 4 Aug 2018, at 18:28, Maxime Beauchemin wrote: > > Let me confirm I'm understanding this right, we're talking specifically > about the CeleryExecutor not starting and `airflow run` (not --raw) > command, and fire up a LocalTaskJob instead? Then we'd still have the > worker

Re: The need for LocalTaskJob

try my forking idea yet, and your proposal is a definite improvement from where we are now. > > B. > > Verstuurd vanaf mijn iPad > >> Op 4 aug. 2018 om 19:40 heeft Ash Berlin-Taylor >> het volgende geschreven: >> >> Comments inline. >> >>&

Re: [VOTE] Airflow 1.10.0rc3

.10) has a timezone in it, and the date is in local time, before it was UTC: before: 2018-07-23T00:00:00/1.log after: 2018-07-23T01:00:00+01:00/1.log We can possibly get away with an updating note about this to set a custom log_filename_template. Testing this now. > On 5 Aug 2018, at 15:00, A

Re: [VOTE] Airflow 1.10.0rc3

-1(binding) from me. Installed with: AIRFLOW_GPL_UNIDECODE=yes pip install 'https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc3/apache-airflow-1.10.0rc3+incubating-bin.tar.gz#egg=apache-airflow[emr, s3, crypto]>=1.10' Install went fine. Our DAGs that use SparkSubmitOperator

Re: [VOTE] Airflow 1.10.0rc3

.10) has a timezone in it, and the date is in local time, before it was UTC: before: 2018-07-23T00:00:00/1.log after: 2018-07-23T01:00:00+01:00/1.log We can possibly get away with an updating note about this to set a custom log_filename_template. Testing this now. > On 5 Aug 2018, at 15:00, A

Re: [VOTE] Airflow 1.10.0rc3

https://github.com/apache/incubator-airflow/compare/master...ashb:updating-for-logging-changes?expand=1. The log format is a bit clunky, but the note about log_task_reader is needed either way. (Do we need a Jira ticket for this sort of change, or is AIRFLOW-XXX okay for this?) > > Cheers &

Re: [VOTE] Airflow 1.10.0rc3

to set the TZ of the connection to UTC in SQLAlchemy to have consistent behaviour? Is this possible some how? I don't know SQLAlchemy that well. -ash > On 5 Aug 2018, at 16:01, Ash Berlin-Taylor > wrote: > > 1.) Missing UPDATING note about change of task_log_reader to now always be

Re: Airflow committers (a list)

- https://github.com/orgs/apache/teams/airflow-committers/members This one at least is populated automatically via a 30-minutely cron job. > On 5 Aug 2018, at 21:35, Sid Anand wrote: > > Committers/Mentors, > We have several

Re: [VOTE] Airflow 1.10.0rc3

e a PR that removes the dependency and implements our own > extension of DateTime. Probably tomorrow. > > Good catch! Just in time :-(. > > B. > >> On 5 Aug 2018, at 22:43, Ash Berlin-Taylor >> wrote: >> >> Entirely possible, though I wasn't e

Re: Airflow-pr merge fails!

I had to change my git remote to use the SSH version - it turns out I had already done that when I was testing the pr tool changes against my fork. https://github.com/apache/incubator-airflow/pull/3680 opened that will correct the remote. Or run this command git remote set-url github

Re: Apache Git Services

I think we now only get them once, rather than once from gitbox, and once again form gitbox sending them to Jira :/ On https://issues.apache.org/jira/browse/INFRA-16854 it was said we require github notifications to go a list (even though we didn't have them before. Guess policies change,

Re: Plan to change type of dag_id from String to Number?

Absolutely - there will still need to be a human-readable DAG id, even we end up with an auto-icrementing integer ID column internally and for table join performance reasons. -ash > On 9 Aug 2018, at 12:35, Юли Волкова wrote: > > How will you understand what your DAG 2 doing enter to it?

Re: [VOTE] Airflow 1.10.0rc4

+0.5 (binding) from me. Tested upgrading form 1.9.0 metadb on Py3.5. Timezones behaving themselves on Postgres. Have not tested the Rbac-based UI. https://github.com/apache/incubator-airflow/commit/d9fecba14c5eb56990508573a91b13ab27ca5153

Re: [VOTE] Airflow 1.10.0rc4

+0.5 (binding) from me. Tested upgrading form 1.9.0 metadb on Py3.5. Timezones behaving themselves on Postgres. Have not tested the Rbac-based UI. https://github.com/apache/incubator-airflow/commit/d9fecba14c5eb56990508573a91b13ab27ca5153

Re: Plan to change type of dag_id from String to Number?

Since this is a big change that would touch much of the code base, before we do this we need to see some hard numbers - timing or benchmarks of queries etc. Also how often do we actually do such a join etc? -ash > On 9 Aug 2018, at 13:04, vardangupta...@gmail.com >

Re: [VOTE] Airflow 1.10.0rc3

t; ``` >>> [testenv] >>> passenv = SLUGIFY_USES_TEXT_UNIDECODE >>> ``` >>> >>> (`setenv` did not work as that provides env vars at runtime but not >>> installtime, as far as I can tell.) >>> >>> >>>

Re: Plan to change type of dag_id from String to Number?

ill >>>> side everyone. If you want id - call it dag_metadata_id and add it. But >>> not >>>> propose change what hasn't backward compatibility. It's to strange. >>>> >>>> On Thu, Aug 9, 2018 at 7:04 AM vardangupta...@gmail.com < >>

Re: apache-airflow v1.10.0 on PyPi?

1.10.0 isn't officially released yet, so that's why it's not on PyPi/tagged yet. (As we are still in the Incubation phase of the project we need our mentors to also sign off on our RC after we, the Airflow community, have voted on it) But yes, we should push the tags to Github. I've done that:

CVE-2017-12614 XSS Vulnerability in Airflow < 1.9

CVE-2017-12614: Apache Reflected Reflected XSS Vulnerability Vendor: The Apache Software Foundation: Versions Affected: < 1.9 Description: It was noticed an XSS in certain 404 pages that could be exploited to perform an XSS attack. Chrome will detect this as a reflected XSS attempt and prevent

Re: Multiple hosts for a single connection

Hmm yes, it appears that the `airflow connections` CLI doesn't let you create multiple connections of the same conn_id. What the WebUI can do the CLI should be able to do also! It should allow that in some way (behind a `--allow-multiple` flag perhaps? I can see an argument for not allowing set

Re: [VOTE] Airflow 1.10.0rc4

> really wants a 1.10.1 quickly so better include it then? Can you make your > vote +1? > > Thx > Bolke > >> On 9 Aug 2018, at 14:06, Ash Berlin-Taylor wrote: >> >> +0.5 (binding) from me. >> >> Tested upgrading form 1.9.0 metadb on Py3.5. Timez

Cloudera Hue in License

Hi everyone, So we include references to Cloudera's Hue in the LICENSE file, and mention it again in the NOTICE file saying: > This product contains a modified portion of 'Hue' developed by Cloudera, Inc. Does anyone know what this refers to? Is it still the case? Grepping for hue doesn't

Re: [RESULT][VOTE] Release Airflow 1.10.0

> >>>>>> >>>>> >>>>> Kaxil Naik >>>>> >>>>> Data Reply >>>>> 2nd Floor, Nova South >>>>> 160 Victoria Street, Westminster >>>>> London SW1E 5LB - UK >>>>> phone: +44

Re: Python 3.6 Support for Airflow 1.10.0

Supporting 3.7 is absolutely something we should do - it just got released while we were already mid-way through the release process of 1.10 and didn't want the scope creep. I'm happy to release a 1.10.1 that supports Py 3.7. The only issue I've seen so far is around the use of `async` as a

Re: Airflow variables and data profiling hidden

Your users are not set up as as Admin users. The mechanism for fixing this depends upon what auth backend you are using. Look in your airflow.cfg under the [core] section for the auth_backend settings, that will (likely) map into one of these classes

Re: Running unit tests against SLUGIFY_USES_TEXT_UNIDECODE and AIRFLOW_GPL_UNIDECODE (also is this broken?)

I don't think we strictly care about running the tests in both these circumstances - it is a flag that control which dep is installed two or three levels down as you say, and the project has it's own tests. I'd rather we spent time on replacing python-nvd with something that means we don't

Re: [RESULT][VOTE] Release Airflow 1.10.0

Could you push the git tag too please Fokko/Bolke? -ash > On 22 Aug 2018, at 08:16, Driesprong, Fokko wrote: > > Thanks Max, > > My PyPI ID is Fokko > > Cheers, Fokko > > 2018-08-21 22:49 GMT+02:00 Maxime Beauchemin : > >> I can, what's your PyPI ID? >> >> Max >> >> On Mon, Aug 20, 2018

Staled Infra ticket to migrate to github - can we chase anyone?

Hi everyone, A while ago (~6 weeks) I opened https://issues.apache.org/jira/browse/INFRA-16602 asking the Apache Infra team to migrate us to github as the primary repo. Sadly nothing has happened in the time, and I tried commenting asking

Re: Retiring Airflow Gitter?

I'm not a fan of slack for open source work - it's a walled garden, signing up is a hurdle (where you have to use a work around of a heroku app), not to mention that the client is just so memory hungry! I'm just a curmudgeon who still likes IRC mainly. So long as I can install

Re: Call for fixes for Airflow 1.10.1

Mail Secure Email. > > ‐‐‐ Original Message ‐‐‐ > On September 3, 2018 3:20 PM, Ash Berlin-Taylor wrote: > >> Hi everyone, >> >> I'm starting the process of gathering fixes for a 1.10.1. So far the list of >> issues I have that we should pull in are >> ht

Re: Call for fixes for Airflow 1.10.1

:22, Ash Berlin-Taylor wrote: > > On 9 September 2018 18:19:40 BST, Bolke de Bruin wrote: > You can already add them to v1-10-test. > > Normally we are a bit cautious to this if you are not the release manager to > ensure that he/she knows what the state is. > > B &

  1   2   >