Re: Sample DB script

Hey Lance, You could definitely write a Python script to do this. MySQL script could be tricky if you want to encrypt the password. Something like: conn = models.Connection( conn_id='airflow_db', conn_type='mysql', host='localhost', login='root', password='', schema='airflow'))

Re: [DISCUSS] Review-then-commit (RTC) vs Commit-then-review (CTR)

Thanks for sharing this information. All, Please share any additional thoughts. In a few days, I will kick off a VOTE thread to decide our process. Mentors, Just curious! Is there an automation we can use to ensure/enforce that PRs and JIRAs receive the +1 binding (from committers) votes needed

Re: boto vs boto3

Great, thanks Chris. I didn't see this on Github and neglected to start looking at Jira. On Wed, May 18, 2016 at 3:14 PM, Chris Riccomini wrote: > Hey David, > > This is tracked here: > > https://issues.apache.org/jira/browse/AIRFLOW-115 > > You'll have to ask Arthur for

PR merge script

Dear team, To make up for being offline for the last week, I have a present for you: https://github.com/apache/incubator-airflow/pull/1515 I started with the script used by the Spark team (and other Apache projects) and medium-modified it to work with Airflow. It makes it super-simple to merge

Re: boto vs boto3

Hey David, This is tracked here: https://issues.apache.org/jira/browse/AIRFLOW-115 You'll have to ask Arthur for the status. Cheers, Chris On Wed, May 18, 2016 at 3:09 PM, David Klosowski wrote: > Hi Airflowers: > > I was testing using the S3KeySensor and it became

boto vs boto3

Hi Airflowers: I was testing using the S3KeySensor and it became clear that it's using boto instead of boto3. Is there any plan to migrate to boto3 (2.0 for instance especially if there is full support for python 3)? https://aws.amazon.com/blogs/aws/now-available-aws-sdk-for-python-3-boto3/

Sample DB script

What is a sample script to add a Connection? Is there a command-line feature or API feature to do this? Or would we just use a MySQL script? -- Lance Norskog lance.nors...@gmail.com Redwood City, CA

Re: Scheduler problems in 1.7?

Ok, we'll update to 1.7.1 when y'all think it's fine. Thanks, Lance Norskog On Wed, May 18, 2016 at 12:01 PM, Bolke de Bruin wrote: > Hey Max, > > Fair point. I’ll make sure that for the next release we jump a bit earlier > on board. We do run integration tests

Re: Scheduler problems in 1.7?

Hey Max, Fair point. I’ll make sure that for the next release we jump a bit earlier on board. We do run integration tests continuously, but only from next month we will reach a certain level of complexity we really will need to start pre-testing releases. - Bolke > Op 18 mei 2016, om 17:49

Re: [DISCUSS] Review-then-commit (RTC) vs Commit-then-review (CTR)

Hey- The consensus approval link is referring to votes (e.g. releases, new committers, new PMCers, etc) rather than code changes. A single +1 is standard for code changes, as Chris describes. However, projects are welcome to provide more fine-grained requirements as well. For example, Hadoop

Re: [DISCUSS] Review-then-commit (RTC) vs Commit-then-review (CTR)

Hey Sid, > In the RTC case, we need 3 +1 binding (a.k.a. committer) votes This sounds very high. Usually one +1 (other than the person sending the) is normal in an RTC scenario. > In the CTR case, we may want a separate develop branch against which to run integration tests and merge to master

Re: Adhoc operators

I think it is a useful feature that nonetheless adds disproportionate complexity -- for example, is there logic for when there is a task downstream from an ad-hoc task, and the ad-hoc task isn't being run? Perhaps there is a way to reengineer it around current Airflow idioms. Maybe we can start

Re: Scheduler problems in 1.7?

There's an RC out that is currently in production at Airbnb (as of Monday) if you want to help us make sure the next version is up fully baked for release. For now Airbnb is carrying most of the risk around deploying new code in production first. Knowing that we don't use all features and

Re: Adhoc operators

The idea there is to be able to ship on-demand tasks along with your DAG. Is it not used because it's not documented? Deprecating may be harder than maintaining it. We'd have to start warning about deprecation in 2.0 soon and add the PR that removes this in the [eventual] 2.0 branch. I don't

Re: Scheduler problems in 1.7?

1.7.1 that most likely will be out at the end of the week, hopefully fixes this indeed. Don't stay on 1.7.0 for too long 1.7.1 contains many stability fixes. Verstuurd vanaf mijn iPad > Op 18 mei 2016 om 03:06 heeft Lance Norskog het > volgende geschreven: > > Has