Hi all, I'm working on the Airflow logging. Got a few PR's in there, but there still a few open ends that I would like to have into 1.9: AIRFLOW-1604 <https://issues.apache.org/jira/browse/AIRFLOW-1604> is mostly cosmetic and almost finished. I think it is important to get AIRFLOW-1611 <https://issues.apache.org/jira/browse/AIRFLOW-1611> in because this will greatly simplify the configuration of the logging. In extend of the these, ticket AIRFLOW-1605 <https://issues.apache.org/jira/browse/AIRFLOW-1605> is also required as this will correctly initialise the log.
Cheers, Fokko 2017-09-13 23:07 GMT+02:00 Chris Riccomini <[email protected]>: > Hey all, > > Figured out the issue with the JIRA list at the top of this email. There's > a bug in the dev/airflow-jira script that limits the results to a max of > 50. Here's the total list of everything that's slated for 1.9.0: > > ISSUE ID |DESCRIPTION |MERGED > AIRFLOW-1608 |GCP Dataflow hook missing pending job state |1 > AIRFLOW-1606 |DAG.sync_to_db is static, but takes a DAG as first|1 > AIRFLOW-1605 |Fix log source of local loggers |0 > AIRFLOW-1602 |Use LoggingMixin for the DAG class |1 > AIRFLOW-1597 |Add GameWisp as Airflow user |1 > AIRFLOW-1594 |Installing via pip copies test files into python l|1 > AIRFLOW-1593 |Expose load_string in WasbHook |1 > AIRFLOW-1586 |MySQL to GCS to BigQuery fails for tables with dat|1 > AIRFLOW-1584 |Remove the insecure /headers endpoints |1 > AIRFLOW-1582 |Improve logging structure of Airflow |1 > AIRFLOW-1580 |Error in string formatter when throwing an excepti|1 > AIRFLOW-1579 |Allow jagged rows in BQ Hook. |1 > AIRFLOW-1577 |Add token support to DatabricksHook |1 > AIRFLOW-1573 |Remove `thrift < 0.10.0` requirement |1 > AIRFLOW-1568 |Add datastore import/export operator |1 > AIRFLOW-1567 |Clean up ML Engine operators |1 > AIRFLOW-1564 |Default logging filename contains a colon |1 > AIRFLOW-1556 |BigQueryBaseCursor should support SQL parameters |1 > AIRFLOW-1546 | add Zymergen to org list in README |1 > AIRFLOW-1535 |Add support for Dataproc serviceAccountScopes in D|1 > AIRFLOW-1529 |Support quoted newlines in Google BigQuery load jo|1 > AIRFLOW-1525 |Fix minor LICENSE & NOTICE issue |0 > AIRFLOW-1522 |Increase size of val column for variable table in |1 > AIRFLOW-1521 |Template fields definition for bigquery_table_dele|1 > AIRFLOW-1507 |Make src, dst and bucket parameters as templated i|1 > AIRFLOW-1505 |Document when Jinja substitution occurs |1 > AIRFLOW-1504 |Log Cluster Name on Dataproc Operator When Execute|1 > AIRFLOW-1499 s|Eliminate duplicate and unneeded code |0 > AIRFLOW-1493 |Fix race condition with airflow run |1 > AIRFLOW-1492 |Add metric for task success/failure |1 > AIRFLOW-1489 |Docs: Typo in BigQueryCheckOperator |1 > AIRFLOW-1478 |Chart -> Owner column should be sortable |1 > AIRFLOW-1476 |Add INSTALL file for source releases |1 > AIRFLOW-1474 |Add dag_id regex for 'airflow clear' CLI command |1 > AIRFLOW-1459 |integration rst doc is broken in github view |1 > AIRFLOW-1438 |Scheduler batch queries should have a limit |1 > AIRFLOW-1437 |BigQueryTableDeleteOperator should define deletion|1 > AIRFLOW-1402 |Cleanup SafeConfigParser DeprecationWarning |1 > AIRFLOW-1401 |Standardize GCP project, region, and zone argument|1 > AIRFLOW-1394 |Add quote_character parameter to GoogleCloudStorag|1 > AIRFLOW-1389 |BigQueryOperator should support `createDisposition|1 > AIRFLOW-1384 |Add ARGO/CaDC |1 > AIRFLOW-1359 |Provide GoogleCloudML operator for model evaluatio|1 > AIRFLOW-1352 |Revert bad logging Handler |0 > AIRFLOW-1350 |Add "query_uri" parameter for Google DataProc oper|1 > AIRFLOW-1345 |Don't commit on each loop |1 > AIRFLOW-1344 |Builds failing on Python 3.5 with AttributeError |1 > AIRFLOW-1343 |Add airflow default label to the dataproc operator|1 > AIRFLOW-1338 |gcp_dataflow_hook is incompatible with the recent |1 > AIRFLOW-1337 |Customize log format via config file |1 > AIRFLOW-1335 |Use buffered logger |1 > AIRFLOW-1333 |Enable copy function for Google Cloud Storage Hook|1 > AIRFLOW-1324 |Make the Druid operator/hook more general |1 > AIRFLOW-1301 |Add New Relic to Airflow user list |1 > AIRFLOW-1299 |Google Dataproc cluster creation operator should s|1 > AIRFLOW-1289 |Don't restrict scheduler threads to CPU cores |1 > AIRFLOW-1286 |BaseTaskRunner - Exception TypeError: a bytes-like|1 > AIRFLOW-1277 |Forbid creation of a known event with empty fields|1 > AIRFLOW-1276 |Forbid event creation with end_data earlier than s|1 > AIRFLOW-1275 |Fix `airflow pool` command exception |1 > AIRFLOW-1274 |HttpSensor parameter params is overriding BaseOper|1 > AIRFLOW-1273 |Google Cloud ML Version and Model CRUD Operator |1 > AIRFLOW-1272 |Google Cloud ML Batch Prediction Operator |1 > AIRFLOW-1271 |Google Cloud ML Training Operator |1 > AIRFLOW-1256 |Add United Airlines as Airflow user |1 > AIRFLOW-1251 |Add eRevalue as an Airflow user |1 > AIRFLOW-1248 |Fix inconsistent configuration name for worker tim|1 > AIRFLOW-1247 |CLI: ignore all dependencies argument ignored |1 > AIRFLOW-1245 |Fix random failure of test_trigger_dag_for_date un|1 > AIRFLOW-1244 |Forbid creation of a pool with empty name |1 > AIRFLOW-1242 |BigQueryHook assumes that a valid project_id can't|1 > AIRFLOW-1237 |Fix IN-predicate sqlalchemy warning |1 > AIRFLOW-1234 |Cover utils.operator_helpers with unit tests |1 > AIRFLOW-1233 |Cover utils.json with unit tests |1 > AIRFLOW-1232 |Remove deprecated readfp warning |1 > AIRFLOW-1231 |Use flask_wtf.CSRFProtect instead of flask_wtf.Csr|1 > AIRFLOW-1221 |Fix DatabricksSubmitRunOperator Templating |1 > AIRFLOW-1217 |Enable logging in Sqoop hook |1 > AIRFLOW-1213 |Add hcatalog parameters to the sqoop operator/hook|1 > AIRFLOW-1208 |Speed-up cli tests |1 > AIRFLOW-1207 |Enable utils.helpers unit tests |1 > AIRFLOW-1203 |Tests failing after oauth upgrade |1 > AIRFLOW-1201 |Update deprecated 'nose-parameterized' library to |1 > AIRFLOW-1198 |HDFSOperator to operate HDFS |0 > AIRFLOW-1197 |Contrib Spark Submit Hook on_kill error |1 > AIRFLOW-1193 |Add Checkr to Airflow user list |1 > AIRFLOW-1191 |Contrib Spark Submit hook should permit override o|1 > AIRFLOW-1189 |Get pandas DataFrame using BigQueryHook fails |1 > AIRFLOW-1188 |Add max_bad_records param to GoogleCloudStorageToB|1 > AIRFLOW-1187 |Obsolete package names in documentation |0 > AIRFLOW-1185 |Incorrect url to PyPi |1 > AIRFLOW-1182 |Contrib Spark Submit operator should template fiel|1 > AIRFLOW-1181 |Enable delete and list function for Google Cloud S|1 > AIRFLOW-1179 |Pandas 0.20 broke Google BigQuery hook |1 > AIRFLOW-1175 |Add Pronto Tools to Airflow user list |1 > AIRFLOW-1173 |Add Robinhood to list of Airflow users |1 > AIRFLOW-1165 |airflow webservice crashes on ubuntu16 - python3 |0 > AIRFLOW-1160 |Upadte SparkSubmitOperator parameters |1 > AIRFLOW-1155 |Add Tails.com to community |1 > AIRFLOW-1149 |Allow custom filters to be added to jinja2 |1 > AIRFLOW-1141 |Remove DAG.crawl_for_tasks method |1 > AIRFLOW-1140 |DatabricksSubmitRunOperator should template the "j|1 > AIRFLOW-1136 |Invalid parameters are not captured for Sqoop oper|1 > AIRFLOW-1125 |Clarify documentation regarding fernet_key |1 > AIRFLOW-1122 |Node strokes are too thin for people with color vi|1 > AIRFLOW-1121 |airflow webserver --pid no longer write out pid fi|0 > AIRFLOW-1118 |Add evo.company to Airflow users |1 > AIRFLOW-1112 |Log which pool is full in scheduler when pool slot|1 > AIRFLOW-1107 |Add support for ftps non-default port |1 > AIRFLOW-1106 |Add Groupalia/Letsbonus |1 > AIRFLOW-1095 |ldap_auth memberOf should come from configuration |1 > AIRFLOW-1094 |Invalid unit tests under `contrib/` |1 > AIRFLOW-1091 |As a release manager I want to be able to compare |1 > AIRFLOW-1090 |Add HBO |1 > AIRFLOW-1089 |Add Spark application arguments to SparkSubmitOper|1 > AIRFLOW-1081 |Task duration page is slow |1 > AIRFLOW-1075 |Cleanup security docs |1 > AIRFLOW-1065 |Add functionality for Azure Blob Storage |1 > AIRFLOW-1059 |Reset_state_for_orphaned_task should operate in ba|1 > AIRFLOW-1058 |Improvements for SparkSubmitOperator |0 > AIRFLOW-1055 |airflow/jobs.py:create_dag_run() exception for @on|0 > AIRFLOW-1051 |Add a test for resetdb to CliTests |1 > AIRFLOW-1047 |Airflow logs vulnerable to XSS |1 > AIRFLOW-1045 |Make log level configurable via airflow.cfg |1 > AIRFLOW-1043 |Documentation issues for operators |1 > AIRFLOW-1041 |DockerOperator replaces its xcom_push method with |1 > AIRFLOW-1040 |Fix typos in comments/docstrings in models.py |1 > AIRFLOW-1036 |Exponential backoff should use randomization |1 > AIRFLOW-1035 |Exponential backoff retry logic should use 2 as ba|1 > AIRFLOW-1034 |Make it possible to connect to S3 in sigv4 regions|1 > AIRFLOW-1030 |HttpHook error when creating HttpSensor |0 > AIRFLOW-1028 |Databricks Operator for Airflow |1 > AIRFLOW-1024 |Handle CeleryExecutor errors gracefully |1 > AIRFLOW-1019 |active_dagruns shouldn't include paused DAGs |0 > AIRFLOW-1018 |Scheduler DAG processes can not log to stdout |0 > AIRFLOW-1016 |Allow HTTP HEAD request method on HTTPSensor |1 > AIRFLOW-1015 |TreeView displayed over task instances |0 > AIRFLOW-1013 |airflow/jobs.py:manage_slas() exception for @once |0 > AIRFLOW-1010 |Add a convenience script for signing |1 > AIRFLOW-1009 |Remove SQLOperator from Concepts page |1 > AIRFLOW-1007 |Jinja sandbox is vulnerable to RCE |1 > AIRFLOW-1006 |Move configuration templates to separate files |1 > AIRFLOW-1005 |Speed up Airflow startup time |1 > AIRFLOW-999 |Support for Redis database |1 > AIRFLOW-997 |Change setup.cfg to point to Apache instead of Max|1 > AIRFLOW-995 |Update Github PR template |1 > AIRFLOW-994 |Add MiNODES to the AIRFLOW Active Users List |1 > AIRFLOW-991 |Mark_success while a task is running leads to fail|0 > AIRFLOW-990 |DockerOperator fails when logging unicode string |1 > AIRFLOW-985 |Extend the sqoop operator/hook with additional par|1 > AIRFLOW-984 |Subdags unrecognized when subclassing SubDagOperat|1 > AIRFLOW-979 |Add GovTech GDS |1 > AIRFLOW-976 |Mark success running task causes it to fail |0 > AIRFLOW-969 |Catch bad python_callable argument at DAG construc|1 > AIRFLOW-963 |Some code examples are not rendered in the airflow|1 > AIRFLOW-960 |Add support for .editorconfig |1 > AIRFLOW-959 |.gitignore file is disorganized and incomplete |1 > AIRFLOW-958 |Improve tooltip readability |1 > AIRFLOW-950 |Missing AWS integrations on documentation::integra|1 > AIRFLOW-947 |Make PrestoHook surface better messages when the P|1 > AIRFLOW-945 |Revert psycopg2 workaround when psycopg2 2.7.1 is |0 > AIRFLOW-943 |Add Digital First Media to the Airflow users list |1 > AIRFLOW-942 |Add mytaxi to Airflow Users |1 > AIRFLOW-935 |Impossible to use plugin executors |1 > AIRFLOW-917 |Incorrectly formatted failure status message |1 > AIRFLOW-916 |Fix ConfigParser deprecation warning |1 > AIRFLOW-914 |Refactor BackfillJobTest.test_backfill_examples to|0 > AIRFLOW-913 |Refactor tests.CoreTest.test_scheduler_job to real|0 > AIRFLOW-912 |Refactor tests and build matrix |0 > AIRFLOW-911 |Add colouring and profiling info on tests |1 > AIRFLOW-903 |Add configuration setting for default DAG view. |1 > AIRFLOW-896 |BigQueryOperator fails to execute with certain inp|1 > AIRFLOW-889 |Minor error in the docstrings for BaseOperator. |1 > AIRFLOW-888 |Operators should not push XComs by default |0 > AIRFLOW-887 |Add compatibility with future v0.16 |1 > AIRFLOW-886 |Pass Operator result to post_execute hook |1 > AIRFLOW-885 |Add Change.org to the list of Airflow users |1 > AIRFLOW-882 |Code example in docs has unnecessary DAG>>Operator|1 > AIRFLOW-881 |Create SubDagOperator within DAG context manager w|1 > AIRFLOW-880 |Fix remote log functionality inconsistencies for W|1 > AIRFLOW-877 |GoogleCloudStorageDownloadOperator: template_ext c|1 > AIRFLOW-875 |Allow HttpSensor params to be templated |1 > AIRFLOW-871 |multiple places use logging.warn() instead of warn|1 > AIRFLOW-866 |Add FTPSensor |1 > AIRFLOW-863 |Example DAG start dates should be recent to avoid |1 > AIRFLOW-862 |Add DaskExecutor |1 > AIRFLOW-860 |Circular module dependency prevents loading of cus|0 > AIRFLOW-854 |Add Open Knowledge International to Airflow users |1 > AIRFLOW-842 |scheduler.clean_dirty raises warning: SAWarning: T|1 > AIRFLOW-840 |Python3 encoding issue in Kerberos |1 > AIRFLOW-836 |The paused and queryview endpoints are vulnerable |1 > AIRFLOW-831 |Fix broken unit tests |1 > AIRFLOW-830 |Plugin manager should log to debug, not info |0 > AIRFLOW-829 |Reduce verbosity of successful Travis unit tests |0 > AIRFLOW-828 |Add maximum size for XComs |0 > AIRFLOW-826 |Add Zendesk Hook |1 > AIRFLOW-825 |Add Dataflow semantics |0 > AIRFLOW-823 |Make task instance details available via API |1 > AIRFLOW-822 |Close the connection before throwing exception in |1 > AIRFLOW-821 |Scheduler dagbag importing not Py3 compatible |1 > AIRFLOW-809 |SqlAlchemy is_ ColumnOperator Causing Errors in MS|0 > AIRFLOW-802 |Integration of spark-submit |0 > AIRFLOW-788 |Context unexpectedly added to hive conf |0 > AIRFLOW-781 |Allow DataFlowJavaOperator to accept jar file stor|1 > AIRFLOW-770 |HDFS hooks should support alternative ways of gett|1 > AIRFLOW-756 |Refactor ssh_hook and ssh_operator |0 > AIRFLOW-751 |SFTP file transfer functionality |1 > AIRFLOW-725 |Make merge tool use OS' keyring for password stora|1 > AIRFLOW-706 |Configuration shell commands are not split properl|1 > AIRFLOW-705 |airflow.configuration.run_command output does not |0 > AIRFLOW-681 |homepage doc link should pointing to apache's repo|1 > AIRFLOW-654 |SSL for AMQP w/ Celery(Executor) |1 > AIRFLOW-645 |HttpHook ignores https |1 > AIRFLOW-365 |Code view in subdag trigger exception |1 > AIRFLOW-300 |Add Google Pubsub hook and operator |1 > > Cheers, > Chris > > On Wed, Sep 13, 2017 at 2:01 PM, Chris Riccomini <[email protected]> > wrote: > > > Note: Everything on master up to the point I cut the branch is included > in > > 1.9.0. I'm not sure what's up with the `dev/airflow-jira compare 1.9.0` > > script.. I'm looking into that now. > > > > On Wed, Sep 13, 2017 at 1:55 PM, Alex Guziel <[email protected]. > > invalid> wrote: > > > >> Nevermind, I misunderstood what you meant. (I thought you meant you were > >> only including things with a fix version of 1.9.0, when you meant master > >> cut + 1.9.0 fix versions) > >> > >> On Wed, Sep 13, 2017 at 1:19 PM, Alex Guziel <[email protected]> > >> wrote: > >> > >> > Shouldn't we include everything on master? > >> > > >> > On Wed, Sep 13, 2017 at 12:45 PM, Chris Riccomini < > >> [email protected]> > >> > wrote: > >> > > >> >> Hey all, > >> >> > >> >> I've cut a 1.9.0 test branch. > >> >> > >> >> https://github.com/apache/incubator-airflow/tree/v1-9-test > >> >> > >> >> Here are the tickets that are being tracked on 1.9.0. > >> >> > >> >> ISSUE ID |DESCRIPTION > |MERGED > >> >> AIRFLOW-1608 |GCP Dataflow hook missing pending job state |1 > >> >> AIRFLOW-1606 |DAG.sync_to_db is static, but takes a DAG as first|1 > >> >> AIRFLOW-1605 |Fix log source of local loggers |0 > >> >> AIRFLOW-1602 |Use LoggingMixin for the DAG class |1 > >> >> AIRFLOW-1597 |Add GameWisp as Airflow user |1 > >> >> AIRFLOW-1594 |Installing via pip copies test files into python l|1 > >> >> AIRFLOW-1593 |Expose load_string in WasbHook |1 > >> >> AIRFLOW-1586 |MySQL to GCS to BigQuery fails for tables with dat|1 > >> >> AIRFLOW-1584 |Remove the insecure /headers endpoints |1 > >> >> AIRFLOW-1582 |Improve logging structure of Airflow |1 > >> >> AIRFLOW-1580 |Error in string formatter when throwing an excepti|1 > >> >> AIRFLOW-1579 |Allow jagged rows in BQ Hook. |1 > >> >> AIRFLOW-1577 |Add token support to DatabricksHook |1 > >> >> AIRFLOW-1573 |Remove `thrift < 0.10.0` requirement |1 > >> >> AIRFLOW-1568 |Add datastore import/export operator |1 > >> >> AIRFLOW-1567 |Clean up ML Engine operators |1 > >> >> AIRFLOW-1564 |Default logging filename contains a colon |1 > >> >> AIRFLOW-1556 |BigQueryBaseCursor should support SQL parameters |1 > >> >> AIRFLOW-1546 | add Zymergen to org list in README |1 > >> >> AIRFLOW-1535 |Add support for Dataproc serviceAccountScopes in D|1 > >> >> AIRFLOW-1529 |Support quoted newlines in Google BigQuery load jo|1 > >> >> AIRFLOW-1522 |Increase size of val column for variable table in |1 > >> >> AIRFLOW-1521 |Template fields definition for bigquery_table_dele|1 > >> >> AIRFLOW-1507 |Make src, dst and bucket parameters as templated i|1 > >> >> AIRFLOW-1505 |Document when Jinja substitution occurs |1 > >> >> AIRFLOW-1504 |Log Cluster Name on Dataproc Operator When Execute|1 > >> >> AIRFLOW-1499ss|Eliminate duplicate and unneeded code |0 > >> >> AIRFLOW-1493 |Fix race condition with airflow run |1 > >> >> AIRFLOW-1492 |Add metric for task success/failure |1 > >> >> AIRFLOW-1489 |Docs: Typo in BigQueryCheckOperator |1 > >> >> AIRFLOW-1478 |Chart -> Owner column should be sortable |1 > >> >> AIRFLOW-1476 |Add INSTALL file for source releases |1 > >> >> AIRFLOW-1474 |Add dag_id regex for 'airflow clear' CLI command |1 > >> >> AIRFLOW-1459 |integration rst doc is broken in github view |1 > >> >> AIRFLOW-1438 |Scheduler batch queries should have a limit |1 > >> >> AIRFLOW-1437 |BigQueryTableDeleteOperator should define deletion|1 > >> >> AIRFLOW-1402 |Cleanup SafeConfigParser DeprecationWarning |1 > >> >> AIRFLOW-1401 |Standardize GCP project, region, and zone argument|1 > >> >> AIRFLOW-1394 |Add quote_character parameter to GoogleCloudStorag|1 > >> >> AIRFLOW-1389 |BigQueryOperator should support `createDisposition|1 > >> >> AIRFLOW-1384 |Add ARGO/CaDC |1 > >> >> AIRFLOW-1359 |Provide GoogleCloudML operator for model evaluatio|1 > >> >> AIRFLOW-1352 |Revert bad logging Handler |0 > >> >> AIRFLOW-1350 |Add "query_uri" parameter for Google DataProc oper|1 > >> >> AIRFLOW-1345 |Don't commit on each loop |1 > >> >> AIRFLOW-1344 |Builds failing on Python 3.5 with AttributeError |1 > >> >> AIRFLOW-1343 |Add airflow default label to the dataproc operator|1 > >> >> AIRFLOW-1338 |gcp_dataflow_hook is incompatible with the recent |1 > >> >> AIRFLOW-1337 |Customize log format via config file |1 > >> >> AIRFLOW-1335 |Use buffered logger |1 > >> >> > >> >> If you have stuff you want to get in, please set it with a fix > version > >> of > >> >> 1.9.0. > >> >> > >> >> Please begin testing, stabilizing, and reporting bugs now! :) > >> >> > >> >> Cheers, > >> >> Chris > >> >> > >> > > >> > > >> > > > > >
