Hello, Apache Airflow (incubating) 1.10.1 (based on RC2) has been accepted. 3 “+1” binding votes received: - Ash Berlin-Taylor (binding) - Kaxil Naik (binding) - Fokko Driesprong (binding)
2 "+1" non-binding votes received: - Deng Xiaodong (non-binding) - Ikar Pohorsky (non-binding) My next step is to open a thread with the IPMC. Cheers, Ash > On 18 Nov 2018, at 16:47, Driesprong, Fokko <fo...@driesprong.frl> wrote: > > A +1 from my side as well. > > Thanks for picking this up Ash. Just checked the new release using Docke > <https://github.com/Fokko/docker-airflow/commit/eb904450ffbc38cee61421ad8c6ff7cfd28c42eb>r, > everything seems to work. > > Cheers, Fokko > > Op za 17 nov. 2018 om 16:43 schreef Deng Xiaodong <xd.den...@gmail.com>: > >> Even though my vote is non-binding, I would like to change my vote to +1 as >> well. >> Reason being the both points I suggested earlier were not regressions from >> 1.10.0, and they should not be blocking the release. >> >> Cheers. >> >> XD >> >> On Sat, Nov 17, 2018 at 8:11 PM Naik Kaxil <k.n...@reply.com> wrote: >> >>> +1 (binding) . I am convinced, we should follow up with 1.10.2 with fixes >>> soon with small number of commits avoiding a huge gap again between minor >>> releases. >>> >>> Regards, >>> Kaxil >>> >>> On 17/11/2018, 11:53, "Ash Berlin-Taylor" <a...@apache.org> wrote: >>> >>> The RBAC UI is still marked as experimental and this isn't a >>> regression from 1.10.0, so could you be convinced to change this to a +1? >>> >>> There are other more critical changes I would like to get out, and I >>> will follow up straight away with a 1.10.2 that addresses this and XD's >>> points. >>> >>> (I feel Bolke's pain :) I'm now moderately annoyed at the Apache >>> release process and how long it takes, it means each release ends up >>> getting big) >>> >>> -ash >>> >>>> >>> >>> Kaxil Naik >>> >>> Data Reply >>> Nova South >>> 160 Victoria Street, Westminster >>> London SW1E 5LB - UK >>> phone: +44 (0)20 7730 6000 >>> k.n...@reply.com >>> www.reply.com >>> On 17 Nov 2018, at 01:01, Naik Kaxil <k.n...@reply.com> wrote: >>>> >>>> -1 (binding) . Tested it on Python 2.7.14, got expected result but >>> had 1 security concern that I want to get in the release. >>>> >>>> Even when 'expose_config'=False, RBAC you still shows the configs >>> which can contain sensitive information like airflow metadb passwords, >> etc. >>>> >>>> If we can get that in +1 from me. The PR with this fixed has been >>> merged in the master, commit: >>> >> https://github.com/apache/incubator-airflow/commit/85abd44e241e17338a800e37a3c2e85ef346898d >>> < >>> >> https://github.com/apache/incubator-airflow/commit/85abd44e241e17338a800e37a3c2e85ef346898d >>>> >>>> >>>> PR: https://github.com/apache/incubator-airflow/pull/4194 < >>> https://github.com/apache/incubator-airflow/pull/4194> >>>> >>>> Regards, >>>> Kaxil >>>> >>>> On 16/11/2018, 13:41, "Deng Xiaodong" <xd.den...@gmail.com >> <mailto: >>> xd.den...@gmail.com>> wrote: >>>> >>>> Hi Ash, >>>> >>>> I would like to give -1 (non-binding), due to two reasons we >>> discussed >>>> earlier on Slack: >>>> >>>> - there is an issue with the new “delete DAG” button in UI. It’s >>> a great >>>> feature, so let’s try to release it “bug-less”. The fix is in PR >>>> https://github.com/apache/incubator-airflow/pull/4069 (But >>> understand your >>>> concern is that this PR comes with no test yet). >>>> >>>> - it may be good to pin all dependencies to a specific version >> to >>> avoid the >>>> incident caused by dependency breaking change (like what happens >>> to Redis >>>> yesterday) >>>> >>>> >>>> Last but not least: nice job! Thanks for your works! >>>> >>>> >>>> XD >>>> >>>> >>>> On Fri, Nov 16, 2018 at 21:13 Ash Berlin-Taylor <a...@apache.org >>> >>> wrote: >>>> >>>>> Friendly reminder for people (and especially committers) to test >>> this out >>>>> and vote on it please! >>>>> >>>>> -ash >>>>> >>>>>> >>>> >>>> Kaxil Naik >>>> >>>> Data Reply >>>> Nova South >>>> 160 Victoria Street, Westminster >>>> London SW1E 5LB - UK >>>> phone: +44 (0)20 7730 6000 >>>> k.n...@reply.com <mailto:k.n...@reply.com> >>>> www.reply.com <http://www.reply.com/> >>>> On 14 Nov 2018, at 22:31, Ash Berlin-Taylor <a...@apache.org >> <mailto: >>> a...@apache.org>> wrote: >>>>>> >>>>>> Hey all, >>>>>> >>>>>> I have cut Airflow 1.10.1 RC2. This email is calling a vote on >> the >>>>> release, which will last for 72 hours. Consider this my (binding) >>> +1. >>>>>> >>>>>> Airflow 1.10.1 RC2 is available at: >>>>>> >>>>>> >>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.1rc2/ >>>>>> >>>>>> apache-airflow-1.10.1rc2+incubating-source.tar.gz is a source >>> release >>>>> that comes with INSTALL instructions. >>>>>> apache-airflow-1.10.1rc2+incubating-bin.tar.gz is the binary >> Python >>>>> "sdist" release. >>>>>> >>>>>> Public keys are available at: >>>>>> >>>>>> >> https://dist.apache.org/repos/dist/release/incubator/airflow/KEYS >>>>>> >>>>>> This release candidate has been published to PyPi as a convince >> for >>>>> testing, but the vote is against the published artefacts at the >>> above URL, >>>>> and not this. To install from PyPI run `pip install --pre >>> apache-airflow` >>>>>> >>>>>> Only votes from PMC members are binding, but members of the >>> community >>>>> are encouraged to test the release and vote with "(non-binding)". >>>>>> >>>>>> Changes since 1.10.1rc1: >>>>>> >>>>>> [AIRFLOW-3343] Update DockerOperator for Docker-py 3.0.0 API >>> changes >>>>> (#4187) >>>>>> [AIRFLOW-XXX] Include 3193 in the changelog >>>>>> [AIRFLOW-XXX] Remove duplicated line in Changelog (#4181) >>>>>> [AIRFLOW-3339] Correctly get DAG timezone when start_date in >>>>> default_args (#4186) >>>>>> >>>>>> Changes since 1.10.1b1: >>>>>> >>>>>> [AIRFLOW-XXX] Correct date and version in Changelog >>>>>> [AIRFLOW-2779] Add license headers to doc files (#4178) >>>>>> [AIRFLOW-XXX] Changelog and version for 1.10.1 >>>>>> [AIRFLOW-2779] Add license headers to doc files (#4178) >>>>>> [AIRFLOW-2779] Add project version to license (#4177) >>>>>> [AIRFLOW-XXX] Sync changelog between release and master branch >>>>>> [AIRFLOW-XXX] Add missing docs for SNS classes (#4155) >>>>>> [AIRFLOW-3072] Assign permission get_logs_with_metadata to viewer >>> role >>>>> (#4175) >>>>>> [AIRFLOW-2723] Update lxml dependancy to >= 4.0.0 >>>>>> [AIRFLOW-3325] Fix UI Page DAGs-column 'Recent Tasks' display >> issue >>>>> (#4173) >>>>>> [AIRFLOW-XXX] Update Updating instructions for changes in 1.10.1 >>>>>> [AIRFLOW-XXX] Fix a few typos in CHANGELOG (#4169) >>>>>> >>>>>> >>>>>> Full changelog is below: >>>>>> >>>>>> New features: >>>>>> >>>>>> [AIRFLOW-2524] Airflow integration with AWS Sagemaker >>>>>> [AIRFLOW-2657] Add ability to delete DAG from web ui >>>>>> [AIRFLOW-2780] Adds IMAP Hook to interact with a mail server >>>>>> [AIRFLOW-2794] Add delete support for Azure blob >>>>>> [AIRFLOW-2912] Add operators for Google Cloud Functions >>>>>> [AIRFLOW-2974] Add Start/Restart/Terminate methods Databricks >> Hook >>>>>> [AIRFLOW-2989] No Parameter to change bootDiskType for >>>>> DataprocClusterCreateOperator >>>>>> [AIRFLOW-3078] Basic operators for Google Compute Engine >>>>>> [AIRFLOW-3147] Update Flask-AppBuilder version >>>>>> [AIRFLOW-3231] Basic operators for Google Cloud SQL (deploy / >>> patch / >>>>> delete) >>>>>> [AIRFLOW-3276] Google Cloud SQL database create / patch / delete >>>>> operators >>>>>> >>>>>> Improvements: >>>>>> >>>>>> [AIRFLOW-393] Add progress callbacks for FTP downloads >>>>>> [AIRFLOW-520] Show Airflow version on web page >>>>>> [AIRFLOW-843] Exceptions now available in context during >>>>> on_failure_callback >>>>>> [AIRFLOW-2476] Update tabulate dependency to v0.8.2 >>>>>> [AIRFLOW-2592] Bump Bleach dependency >>>>>> [AIRFLOW-2622] Add "confirm=False" option to SFTPOperator >>>>>> [AIRFLOW-2662] support affinity & nodeSelector policies for >>> kubernetes >>>>> executor/operator >>>>>> [AIRFLOW-2709] Improve error handling in Databricks hook >>>>>> [AIRFLOW-2723] Update lxml dependancy to >= 4.0. >>>>>> [AIRFLOW-2763] No precheck mechanism in place during worker >>>>> initialisation for the connection to metadata database >>>>>> [AIRFLOW-2789] Add ability to create single node cluster to >>>>> DataprocClusterCreateOperator >>>>>> [AIRFLOW-2797] Add ability to create Google Dataproc cluster with >>> custom >>>>> image >>>>>> [AIRFLOW-2854] kubernetes_pod_operator add more configuration >> items >>>>>> [AIRFLOW-2855] Need to Check Validity of Cron Expression When >>> Process >>>>> DAG File/Zip File >>>>>> [AIRFLOW-2904] Clean an unnecessary line in >>>>> airflow/executors/celery_executor.py >>>>>> [AIRFLOW-2921] A trivial incorrectness in CeleryExecutor() >>>>>> [AIRFLOW-2922] Potential deal-lock bug in CeleryExecutor() >>>>>> [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file >>>>>> [AIRFLOW-2949] Syntax Highlight for Single Quote >>>>>> [AIRFLOW-2951] dag_run end_date Null after a dag is finished >>>>>> [AIRFLOW-2956] Kubernetes tolerations for pod operator >>>>>> [AIRFLOW-2997] Support for clustered tables in Bigquery >>> hooks/operators >>>>>> [AIRFLOW-3006] Fix error when schedule_interval="None" >>>>>> [AIRFLOW-3008] Move Kubernetes related example DAGs to >>>>> contrib/example_dags >>>>>> [AIRFLOW-3025] Allow to specify dns and dns-search parameters for >>>>> DockerOperator >>>>>> [AIRFLOW-3067] (www_rbac) Flask flash messages are not displayed >>>>> properly (no background color) >>>>>> [AIRFLOW-3069] Decode output of S3 file transform operator >>>>>> [AIRFLOW-3072] Assign permission get_logs_with_metadata to viewer >>> role >>>>>> [AIRFLOW-3090] INFO logs are too verbose >>>>>> [AIRFLOW-3103] Update Flask-Login >>>>>> [AIRFLOW-3112] Align SFTP hook with SSH hook >>>>>> [AIRFLOW-3119] Enable loglevel on celery worker and inherit from >>>>> airflow.cfg >>>>>> [AIRFLOW-3137] Make ProxyFix middleware optional >>>>>> [AIRFLOW-3173] Add _cmd options for more password config options >>>>>> [AIRFLOW-3177] Change scheduler_heartbeat metric from gauge to >>> counter >>>>>> [AIRFLOW-3195] Druid Hook: Log ingestion spec and task id >>>>>> [AIRFLOW-3197] EMR Hook is missing some parameters to valid on >> the >>> AWS >>>>> API >>>>>> [AIRFLOW-3232] Make documentation for GCF Functions operator more >>>>> readable >>>>>> [AIRFLOW-3262] Can't get log containing Response when using >>>>> SimpleHttpOperator >>>>>> [AIRFLOW-3265] Add support for "unix_socket" in connection extra >>> for >>>>> Mysql Hook >>>>>> >>>>>> Doc-only changes: >>>>>> >>>>>> [AIRFLOW-1441] Tutorial Inconsistencies Between Example Pipeline >>>>> Definition and Recap >>>>>> [AIRFLOW-2682] Add how-to guide(s) for how to use basic operators >>> like >>>>> BashOperator and PythonOperator >>>>>> [AIRFLOW-3104] .airflowignore feature is not mentioned at all in >>>>> documentation >>>>>> [AIRFLOW-3237] Refactor example DAGs >>>>>> [AIRFLOW-3187] Update airflow.gif file with a slower version >>>>>> [AIRFLOW-3159] Update Airflow documentation on GCP Logging >>>>>> [AIRFLOW-3030] Command Line docs incorrect subdir >>>>>> [AIRFLOW-2990] Docstrings for Hooks/Operators are in incorrect >>> format >>>>>> [AIRFLOW-3127] Celery SSL Documentation is out-dated >>>>>> [AIRFLOW-2779] Add license headers to doc files >>>>>> [AIRFLOW-2779] Add project version to license >>>>>> >>>>>> Bug fixes: >>>>>> >>>>>> [AIRFLOW-839] docker_operator.py attempts to log status key >> without >>>>> first checking existence >>>>>> [AIRFLOW-1104] Concurrency check in scheduler should count queued >>> tasks >>>>> as well as running >>>>>> [AIRFLOW-1163] Add support for x-forwarded-* headers to support >>> access >>>>> behind AWS ELB >>>>>> [AIRFLOW-1195] Cleared tasks in SubDagOperator do not trigger >>> Parent >>>>> dag_runs >>>>>> [AIRFLOW-1508] Skipped state not part of State.task_states >>>>>> [AIRFLOW-1762] Use key_file in SSHHook.create_tunnel() >>>>>> [AIRFLOW-1837] Differing start_dates on tasks not respected by >>> scheduler. >>>>>> [AIRFLOW-1874] Support standard SQL in Check, ValueCheck and >>>>> IntervalCheck BigQuery operators >>>>>> [AIRFLOW-1917] print() from python operators end up with extra >> new >>> line >>>>>> [AIRFLOW-1970] Database cannot be initialized if an invalid >> fernet >>> key >>>>> is provided >>>>>> [AIRFLOW-2145] Deadlock after clearing a running task >>>>>> [AIRFLOW-2216] Cannot specify a profile for AWS Hook to load with >>> s3 >>>>> config file >>>>>> [AIRFLOW-2574] initdb fails when mysql password contains percent >>> sign >>>>>> [AIRFLOW-2707] Error accessing log files from web UI >>>>>> [AIRFLOW-2716] Replace new Python 3.7 keywords >>>>>> [AIRFLOW-2744] RBAC app doesn't integrate plugins (blueprints >> etc) >>>>>> [AIRFLOW-2772] BigQuery hook does not allow specifying both the >>>>> partition field name and table name at the same time >>>>>> [AIRFLOW-2778] Bad Import in collect_dag in DagBag >>>>>> [AIRFLOW-2786] Variables view fails to render if a variable has >> an >>> empty >>>>> key >>>>>> [AIRFLOW-2799] Filtering UI objects by datetime is broken >>>>>> [AIRFLOW-2800] Remove airflow/ low-hanging linting errors >>>>>> [AIRFLOW-2825] S3ToHiveTransfer operator may not may able to >>> handle GZIP >>>>> file with uppercase ext in S3 >>>>>> [AIRFLOW-2848] dag_id is missing in metadata table "job" for >>> LocalTaskJob >>>>>> [AIRFLOW-2860] DruidHook: time variable is not updated correctly >>> when >>>>> checking for timeout >>>>>> [AIRFLOW-2865] Race condition between on_success_callback and >>>>> LocalTaskJob's cleanup >>>>>> [AIRFLOW-2893] Stuck dataflow job due to jobName mismatch. >>>>>> [AIRFLOW-2895] Prevent scheduler from spamming heartbeats/logs >>>>>> [AIRFLOW-2900] Code not visible for Packaged DAGs >>>>>> [AIRFLOW-2905] Switch to regional dataflow job service. >>>>>> [AIRFLOW-2907] Sendgrid - Attachments - ERROR - Object of type >>> 'bytes' >>>>> is not JSON serializable >>>>>> [AIRFLOW-2938] Invalid 'extra' field in connection can raise an >>>>> AttributeError when attempting to edit >>>>>> [AIRFLOW-2979] Deprecated Celery Option not in Options list >>>>>> [AIRFLOW-2981] TypeError in dataflow operators when using GCS jar >>> or >>>>> py_file >>>>>> [AIRFLOW-2984] Cannot convert naive_datetime when task has a >> naive >>>>> start_date/end_date >>>>>> [AIRFLOW-2994] flatten_results in BigQueryOperator/BigQueryHook >>> should >>>>> default to None >>>>>> [AIRFLOW-3002] ValueError in dataflow operators when using GCS >> jar >>> or >>>>> py_file >>>>>> [AIRFLOW-3012] Email on sla miss is send only to first address on >>> the >>>>> list >>>>>> [AIRFLOW-3046] ECS Operator mistakenly reports success when task >> is >>>>> killed due to EC2 host termination >>>>>> [AIRFLOW-3064] No output from `airflow test` due to default >> logging >>>>> config >>>>>> [AIRFLOW-3072] Only admin can view logs in RBAC UI >>>>>> [AIRFLOW-3079] Improve initdb to support MSSQL Server >>>>>> [AIRFLOW-3089] Google auth doesn't work under http >>>>>> [AIRFLOW-3099] Errors raised when some blocs are missing in >>> airflow.cfg >>>>>> [AIRFLOW-3109] Default user permission should contain 'can_clear' >>>>>> [AIRFLOW-3111] Confusing comments and instructions for log >>> templates in >>>>> UPDATING.md and default_airflow.cfg >>>>>> [AIRFLOW-3124] Broken webserver debug mode (RBAC) >>>>>> [AIRFLOW-3136] Scheduler Failing the Task retries run while >>> processing >>>>> Executor Events >>>>>> [AIRFLOW-3138] Migration cc1e65623dc7 creates issues with >> postgres >>>>>> [AIRFLOW-3161] Log Url link does not link to task instance logs >> in >>> RBAC >>>>> UI >>>>>> [AIRFLOW-3162] HttpHook fails to parse URL when port is specified >>>>>> [AIRFLOW-3183] Potential Bug in >>>>> utils/dag_processing/DagFileProcessorManager.max_runs_reached() >>>>>> [AIRFLOW-3203] Bugs in DockerOperator & Some operator test >> scripts >>> were >>>>> named incorrectly >>>>>> [AIRFLOW-3238] Dags, removed from the filesystem, are not >>> deactivated on >>>>> initdb >>>>>> [AIRFLOW-3268] Cannot pass SSL dictionary to mysql connection via >>> URL >>>>>> [AIRFLOW-3277] Invalid timezone transition handling for cron >>> schedules >>>>>> [AIRFLOW-3295] Require encryption in DaskExecutor when >>> certificates are >>>>> configured. >>>>>> [AIRFLOW-3297] EmrStepSensor marks cancelled step as successful >>> >>> >>> >>> >>