[GitHub] [airflow] galuszkak commented on issue #4772: [AIRFLOW-3937] KubernetesPodOperator support for envFrom configMapRef…
galuszkak commented on issue #4772: [AIRFLOW-3937] KubernetesPodOperator support for envFrom configMapRef… URL: https://github.com/apache/airflow/pull/4772#issuecomment-474695251 @ashb can I understand reasoning behind that, why we want to test that now? None of the tests were testing output of KubernetesRequestFactory before, only classes like Pod/PodLauncher to check if fields there are correct. If we want to start testing that, I believe there should be separate ticket to make sure we cover most of the cases of outputs of KubernetesRequestFactory that is actually producing what we want in every case. Because for now there isn't single assertion on product of that class, only abstraction on top of it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4123) Add Exception handling for _change_state method in K8 Executor
[ https://issues.apache.org/jira/browse/AIRFLOW-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796817#comment-16796817 ] ASF GitHub Bot commented on AIRFLOW-4123: - dimberman commented on pull request #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 … URL: https://github.com/apache/airflow/pull/4941 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Exception handling for _change_state method in K8 Executor > -- > > Key: AIRFLOW-4123 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4123 > Project: Apache Airflow > Issue Type: Bug >Reporter: Anand >Assignee: Anand >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4123) Add Exception handling for _change_state method in K8 Executor
[ https://issues.apache.org/jira/browse/AIRFLOW-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796818#comment-16796818 ] ASF subversion and git services commented on AIRFLOW-4123: -- Commit bac8244e0a177f3e016b08d0ddbe3e8696424982 in airflow's branch refs/heads/master from andy-g14 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=bac8244 ] [AIRFLOW-4123] Add Exception handling for _change_state method in K8 Executor (#4941) > Add Exception handling for _change_state method in K8 Executor > -- > > Key: AIRFLOW-4123 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4123 > Project: Apache Airflow > Issue Type: Bug >Reporter: Anand >Assignee: Anand >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] dimberman merged pull request #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 …
dimberman merged pull request #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 … URL: https://github.com/apache/airflow/pull/4941 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-4130) Enhance hook of HivePartitionSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noah updated AIRFLOW-4130: -- Description: h2. Backgrounds HivePartitionSensor is useful to wait a partition to be completed. HivePartitionSensor uses HiveMetastoreHook to check the existence of partition. HiveMetastoreHook invokes [get_partitions_by_filter|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L6234-L6260] which contains additional information of Partition. However, it seems that there is a alternative method [HiveMetaStore#get_num_partitions_by_filter|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L6335-L6357] which only retrieve the number of partitions for given filter. h2. As Is [HivePartitionSensor|https://github.com/apache/airflow/blob/master/airflow/sensors/hive_partition_sensor.py] * method 'poke' ** It invokes check_for_partition of hook which uses get_partitions_by_filter h2. To be [HivePartitionSensor|https://github.com/apache/airflow/blob/master/airflow/sensors/hive_partition_sensor.py] * Constructor ** additional arguments 'num_partitions=1' (default) * method 'poke' ** It will call 'get_num_partitions' of HiveMetastoreHook and compare the return value with given num_partitions [HiveMetastoreHook|https://github.com/apache/airflow/blob/master/airflow/hooks/hive_hooks.py] * Add new method get_num_partitions(self, schema, table_name, filter=None) ** It will invoke get_num_partitions_by_filter of ThriftHiveMetastore *** [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift#L2112-L2113] was: h2. Backgrounds HivePartitionSensor is useful to wait a partition to be completed. HivePartitionSensor uses HiveMetastoreHook to check the existence of partition. HiveMetastoreHook invokes [get_partitions_by_filter|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L6234-L6260] which contains additional information of Partition. It seems that there is a alternative method [get_num_partitions_by_filter|[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L6335-L6357]|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L6335-L6357],] h2. As Is [HivePartitionSensor|[https://github.com/apache/airflow/blob/master/airflow/sensors/hive_partition_sensor.py]] * method 'poke' ** It invokes check_for_partition of hook which uses get_partitions_by_filter h2. To be [HivePartitionSensor|[https://github.com/apache/airflow/blob/master/airflow/sensors/hive_partition_sensor.py]] * Constructor ** additional arguments 'num_partitions=1' (default) * method 'poke' ** It will call 'get_num_partitions' of HiveMetastoreHook and compare the return value with given num_partitions [HiveMetastoreHook|[https://github.com/apache/airflow/blob/master/airflow/hooks/hive_hooks.py]] * Add new method get_num_partitions(self, schema, table_name, filter=None) ** It will invoke get_num_partitions_by_filter of ThriftHiveMetastore *** [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift#L2112-L2113] ** > Enhance hook of HivePartitionSensor > --- > > Key: AIRFLOW-4130 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4130 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Noah >Assignee: Noah >Priority: Minor > > h2. Backgrounds > HivePartitionSensor is useful to wait a partition to be completed. > HivePartitionSensor uses HiveMetastoreHook to check the existence of > partition. > HiveMetastoreHook invokes > [get_partitions_by_filter|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L6234-L6260] > which contains additional information of Partition. However, it seems that > there is a alternative method > [HiveMetaStore#get_num_partitions_by_filter|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L6335-L6357] > which only retrieve the number of partitions for given filter. > h2. As Is > > [HivePartitionSensor|https://github.com/apache/airflow/blob/master/airflow/sensors/hive_partition_sensor.py] > > * method 'poke' > ** It invokes check_for_partition of hook which uses
[GitHub] [airflow] milton0825 commented on issue #4946: [AIRFLOW-4118] instrument DagRun duration
milton0825 commented on issue #4946: [AIRFLOW-4118] instrument DagRun duration URL: https://github.com/apache/airflow/pull/4946#issuecomment-474681782 PTAL @feng-tao @Fokko tests passed on local CI: https://travis-ci.org/milton0825/airflow/builds/508680332 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-4130) Enhance hook of HivePartitionSensor
Noah created AIRFLOW-4130: - Summary: Enhance hook of HivePartitionSensor Key: AIRFLOW-4130 URL: https://issues.apache.org/jira/browse/AIRFLOW-4130 Project: Apache Airflow Issue Type: New Feature Reporter: Noah Assignee: Noah h2. Backgrounds HivePartitionSensor is useful to wait a partition to be completed. HivePartitionSensor uses HiveMetastoreHook to check the existence of partition. HiveMetastoreHook invokes [get_partitions_by_filter|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L6234-L6260] which contains additional information of Partition. It seems that there is a alternative method [get_num_partitions_by_filter|[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L6335-L6357]|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L6335-L6357],] h2. As Is [HivePartitionSensor|[https://github.com/apache/airflow/blob/master/airflow/sensors/hive_partition_sensor.py]] * method 'poke' ** It invokes check_for_partition of hook which uses get_partitions_by_filter h2. To be [HivePartitionSensor|[https://github.com/apache/airflow/blob/master/airflow/sensors/hive_partition_sensor.py]] * Constructor ** additional arguments 'num_partitions=1' (default) * method 'poke' ** It will call 'get_num_partitions' of HiveMetastoreHook and compare the return value with given num_partitions [HiveMetastoreHook|[https://github.com/apache/airflow/blob/master/airflow/hooks/hive_hooks.py]] * Add new method get_num_partitions(self, schema, table_name, filter=None) ** It will invoke get_num_partitions_by_filter of ThriftHiveMetastore *** [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift#L2112-L2113] ** -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] milton0825 commented on issue #4898: [AIRFLOW-XXXX] how to setup simple test env with mysql
milton0825 commented on issue #4898: [AIRFLOW-] how to setup simple test env with mysql URL: https://github.com/apache/airflow/pull/4898#issuecomment-474672943 Originally I put it in the how-to guide but @mik-laj suggest that I should put it in the contribute.md This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #4947: [AIRFLOW-3706] Fix tooltip max-width by correcting ordering of CSS files
codecov-io commented on issue #4947: [AIRFLOW-3706] Fix tooltip max-width by correcting ordering of CSS files URL: https://github.com/apache/airflow/pull/4947#issuecomment-474670207 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4947?src=pr=h1) Report > Merging [#4947](https://codecov.io/gh/apache/airflow/pull/4947?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a8f037e6de6bb30ac00383a426012b077858c3a1?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4947?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4947 +/- ## == + Coverage 75.58% 75.59% +<.01% == Files 454 454 Lines 2919729203 +6 == + Hits2206922075 +6 Misses 7128 7128 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4947?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/hooks/aws\_glue\_catalog\_hook.py](https://codecov.io/gh/apache/airflow/pull/4947/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F3c19nbHVlX2NhdGFsb2dfaG9vay5weQ==) | `100% <0%> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4947?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4947?src=pr=footer). Last update [a8f037e...4b1357f](https://codecov.io/gh/apache/airflow/pull/4947?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #4898: [AIRFLOW-XXXX] how to setup simple test env with mysql
feng-tao commented on issue #4898: [AIRFLOW-] how to setup simple test env with mysql URL: https://github.com/apache/airflow/pull/4898#issuecomment-474667928 Any specific reason that this doc is in contribution.md not in the howto(https://github.com/apache/airflow/tree/master/docs/howto)? Currently Airflow has couples of sqlORM to support which seems a bit ODD to put Mysql here (not postgres or sqlite or others etc) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4106) Instrument starving tasks in pool
[ https://issues.apache.org/jira/browse/AIRFLOW-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796738#comment-16796738 ] ASF GitHub Bot commented on AIRFLOW-4106: - feng-tao commented on pull request #4927: [AIRFLOW-4106] instrument starving tasks in pool URL: https://github.com/apache/airflow/pull/4927 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Instrument starving tasks in pool > - > > Key: AIRFLOW-4106 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4106 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4106) Instrument starving tasks in pool
[ https://issues.apache.org/jira/browse/AIRFLOW-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796739#comment-16796739 ] ASF subversion and git services commented on AIRFLOW-4106: -- Commit e5943460b15dfd6d1fdc47e2025005e7dab41a92 in airflow's branch refs/heads/master from Chao-Han Tsai [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e594346 ] [AIRFLOW-4106] instrument staving tasks in pool (#4927) > Instrument starving tasks in pool > - > > Key: AIRFLOW-4106 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4106 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4106) Instrument starving tasks in pool
[ https://issues.apache.org/jira/browse/AIRFLOW-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4106. --- Resolution: Fixed Fix Version/s: 1.10.3 > Instrument starving tasks in pool > - > > Key: AIRFLOW-4106 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4106 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 1.10.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] feng-tao merged pull request #4927: [AIRFLOW-4106] instrument starving tasks in pool
feng-tao merged pull request #4927: [AIRFLOW-4106] instrument starving tasks in pool URL: https://github.com/apache/airflow/pull/4927 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-4122) Remove chain function in airflow.utils
[ https://issues.apache.org/jira/browse/AIRFLOW-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4122. --- Resolution: Fixed > Remove chain function in airflow.utils > -- > > Key: AIRFLOW-4122 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4122 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: patch > Fix For: 1.10.3 > > > I had submit [pr|[https://github.com/apache/airflow/pull/4779]] and it > related to Jira ticket . > In PR comment, I was suggest close this PR and submit new one to remove > function `chain` in `airflow.utils.heplers`, due to Airflower think use bit > operation like `>>` or `<<` visual and easier to explain, and have multiple > ways is confusion to users. BTW, this function was created 3 year ago and > never get updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] codecov-io edited a comment on issue #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 …
codecov-io edited a comment on issue #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 … URL: https://github.com/apache/airflow/pull/4941#issuecomment-474257350 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4941?src=pr=h1) Report > Merging [#4941](https://codecov.io/gh/apache/airflow/pull/4941?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a8f037e6de6bb30ac00383a426012b077858c3a1?src=pr=desc) will **decrease** coverage by `0.59%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/airflow/pull/4941?src=pr=tree) ```diff @@Coverage Diff@@ ## master#4941 +/- ## = - Coverage 75.58% 74.99% -0.6% = Files 454 454 Lines 2919729201 +4 = - Hits2206921899-170 - Misses 7128 7302+174 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4941?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/4941/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4ZWN1dG9ycy9rdWJlcm5ldGVzX2V4ZWN1dG9yLnB5) | `64.43% <0%> (-0.8%)` | :arrow_down: | | [airflow/contrib/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/4941/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvdm9sdW1lX21vdW50LnB5) | `33.33% <0%> (-66.67%)` | :arrow_down: | | [airflow/contrib/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/4941/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvcG9kX2xhdW5jaGVyLnB5) | `38.21% <0%> (-52.04%)` | :arrow_down: | | [airflow/contrib/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/4941/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMva3ViZV9jbGllbnQucHk=) | `27.58% <0%> (-51.73%)` | :arrow_down: | | [airflow/contrib/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/4941/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvdm9sdW1lLnB5) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/contrib/kubernetes/pod\_generator.py](https://codecov.io/gh/apache/airflow/pull/4941/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvcG9kX2dlbmVyYXRvci5weQ==) | `43.24% <0%> (-43.25%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/4941/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `58.33% <0%> (-40.28%)` | :arrow_down: | | [.../kubernetes\_request\_factory/pod\_request\_factory.py](https://codecov.io/gh/apache/airflow/pull/4941/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMva3ViZXJuZXRlc19yZXF1ZXN0X2ZhY3RvcnkvcG9kX3JlcXVlc3RfZmFjdG9yeS5weQ==) | `62.06% <0%> (-37.94%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/4941/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.51% <0%> (-4.9%)` | :arrow_down: | | [...etes\_request\_factory/kubernetes\_request\_factory.py](https://codecov.io/gh/apache/airflow/pull/4941/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMva3ViZXJuZXRlc19yZXF1ZXN0X2ZhY3Rvcnkva3ViZXJuZXRlc19yZXF1ZXN0X2ZhY3RvcnkucHk=) | `68.42% <0%> (-4.22%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4941?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4941?src=pr=footer). Last update [a8f037e...c6fac9f](https://codecov.io/gh/apache/airflow/pull/4941?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4122) Remove chain function in airflow.utils
[ https://issues.apache.org/jira/browse/AIRFLOW-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796715#comment-16796715 ] ASF GitHub Bot commented on AIRFLOW-4122: - feng-tao commented on pull request #4940: [AIRFLOW-4122] Remove chain function URL: https://github.com/apache/airflow/pull/4940 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove chain function in airflow.utils > -- > > Key: AIRFLOW-4122 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4122 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: patch > Fix For: 1.10.3 > > > I had submit [pr|[https://github.com/apache/airflow/pull/4779]] and it > related to Jira ticket . > In PR comment, I was suggest close this PR and submit new one to remove > function `chain` in `airflow.utils.heplers`, due to Airflower think use bit > operation like `>>` or `<<` visual and easier to explain, and have multiple > ways is confusion to users. BTW, this function was created 3 year ago and > never get updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4122) Remove chain function in airflow.utils
[ https://issues.apache.org/jira/browse/AIRFLOW-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796717#comment-16796717 ] ASF subversion and git services commented on AIRFLOW-4122: -- Commit ee71a8bb102dcc3a591c5c175ab88a9043cffb0f in airflow's branch refs/heads/master from zhongjiajie [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ee71a8b ] [AIRFLOW-4122] Remove chain function (#4940) * [AIRFLOW-4122] Remove chain function Bit operation like `>>` or `<<` are suggested to set dependency, which visual and easier to explain. and have multiple ways is confusion * change UPDATING.md as recommend [ci skip] > Remove chain function in airflow.utils > -- > > Key: AIRFLOW-4122 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4122 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: patch > Fix For: 1.10.3 > > > I had submit [pr|[https://github.com/apache/airflow/pull/4779]] and it > related to Jira ticket . > In PR comment, I was suggest close this PR and submit new one to remove > function `chain` in `airflow.utils.heplers`, due to Airflower think use bit > operation like `>>` or `<<` visual and easier to explain, and have multiple > ways is confusion to users. BTW, this function was created 3 year ago and > never get updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4122) Remove chain function in airflow.utils
[ https://issues.apache.org/jira/browse/AIRFLOW-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796716#comment-16796716 ] ASF subversion and git services commented on AIRFLOW-4122: -- Commit ee71a8bb102dcc3a591c5c175ab88a9043cffb0f in airflow's branch refs/heads/master from zhongjiajie [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ee71a8b ] [AIRFLOW-4122] Remove chain function (#4940) * [AIRFLOW-4122] Remove chain function Bit operation like `>>` or `<<` are suggested to set dependency, which visual and easier to explain. and have multiple ways is confusion * change UPDATING.md as recommend [ci skip] > Remove chain function in airflow.utils > -- > > Key: AIRFLOW-4122 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4122 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: patch > Fix For: 1.10.3 > > > I had submit [pr|[https://github.com/apache/airflow/pull/4779]] and it > related to Jira ticket . > In PR comment, I was suggest close this PR and submit new one to remove > function `chain` in `airflow.utils.heplers`, due to Airflower think use bit > operation like `>>` or `<<` visual and easier to explain, and have multiple > ways is confusion to users. BTW, this function was created 3 year ago and > never get updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] feng-tao merged pull request #4940: [AIRFLOW-4122] Remove chain function
feng-tao merged pull request #4940: [AIRFLOW-4122] Remove chain function URL: https://github.com/apache/airflow/pull/4940 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #4940: [AIRFLOW-4122] Remove chain function
feng-tao commented on issue #4940: [AIRFLOW-4122] Remove chain function URL: https://github.com/apache/airflow/pull/4940#issuecomment-474659183 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] zhongjiajie commented on issue #4940: [AIRFLOW-4122] Remove chain function
zhongjiajie commented on issue #4940: [AIRFLOW-4122] Remove chain function URL: https://github.com/apache/airflow/pull/4940#issuecomment-474658226 @feng-tao I change `UPDATING.md` and skip CI, PTAL thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] zhongjiajie commented on a change in pull request #4940: [AIRFLOW-4122] Remove chain function
zhongjiajie commented on a change in pull request #4940: [AIRFLOW-4122] Remove chain function URL: https://github.com/apache/airflow/pull/4940#discussion_r267164811 ## File path: UPDATING.md ## @@ -24,6 +24,11 @@ assists users migrating to a new version. ## Airflow Master +### Deprecation chain function after Airflow 1.10.2 + +Bit operation like `>>` or `<<` are suggested to set dependency, which visual and easier to explain. Review comment: I will change it. thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] zhongjiajie commented on a change in pull request #4940: [AIRFLOW-4122] Remove chain function
zhongjiajie commented on a change in pull request #4940: [AIRFLOW-4122] Remove chain function URL: https://github.com/apache/airflow/pull/4940#discussion_r267164655 ## File path: UPDATING.md ## @@ -24,6 +24,11 @@ assists users migrating to a new version. ## Airflow Master +### Deprecation chain function after Airflow 1.10.2 Review comment: Agree with that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on a change in pull request #4940: [AIRFLOW-4122] Remove chain function
feng-tao commented on a change in pull request #4940: [AIRFLOW-4122] Remove chain function URL: https://github.com/apache/airflow/pull/4940#discussion_r267163296 ## File path: UPDATING.md ## @@ -24,6 +24,11 @@ assists users migrating to a new version. ## Airflow Master +### Deprecation chain function after Airflow 1.10.2 + +Bit operation like `>>` or `<<` are suggested to set dependency, which visual and easier to explain. Review comment: WDYT This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on a change in pull request #4940: [AIRFLOW-4122] Remove chain function
feng-tao commented on a change in pull request #4940: [AIRFLOW-4122] Remove chain function URL: https://github.com/apache/airflow/pull/4940#discussion_r267163273 ## File path: UPDATING.md ## @@ -24,6 +24,11 @@ assists users migrating to a new version. ## Airflow Master +### Deprecation chain function after Airflow 1.10.2 Review comment: WDYT This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman commented on issue #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 …
dimberman commented on issue #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 … URL: https://github.com/apache/airflow/pull/4941#issuecomment-474655058 @feng-tao Ahh good point! I've rerun the one failing test (this one seems to be around minikube not working correctly) and will merge if this one passes. @ashb it doesn't seem that my jira perms have come in yet. Are there further steps I need to take or is that just a migration? (thank you to you both for helping me get the hang of this btw :) ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] XD-DENG commented on issue #4947: [AIRFLOW-3706] Fix tooltip max-width by correcting ordering of CSS files
XD-DENG commented on issue #4947: [AIRFLOW-3706] Fix tooltip max-width by correcting ordering of CSS files URL: https://github.com/apache/airflow/pull/4947#issuecomment-474651753 I have the same impression as @kaxil , because we did encounter this issue in the releasing process of 1.10.2. It should have been resolved by https://github.com/apache/airflow/pull/3687/files#diff-45f26f49fdf6bbac6cc050c125b8f6f6R41 . If we still have this issue in master branch, not sure if it's because of the `www/www_rbac` switching. But I think there is no harm from the change in this PR, and I'm ok to merge this. @kaxil do you agree? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #4947: [AIRFLOW-3706] Fix tooltip max-width by correcting ordering of CSS files
kaxil commented on issue #4947: [AIRFLOW-3706] Fix tooltip max-width by correcting ordering of CSS files URL: https://github.com/apache/airflow/pull/4947#issuecomment-474648784 This is strange as it was resolved as a part of https://github.com/apache/airflow/pull/3687 https://github.com/apache/airflow/pull/4539#issuecomment-454733779 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jmcarp commented on a change in pull request #4926: [AIRFLOW-4104] Add type annotations to common classes.
jmcarp commented on a change in pull request #4926: [AIRFLOW-4104] Add type
annotations to common classes.
URL: https://github.com/apache/airflow/pull/4926#discussion_r267152200
##
File path: airflow/models/connection.py
##
@@ -251,6 +251,7 @@ def get_hook(self):
elif self.conn_type == 'grpc':
from airflow.contrib.hooks.grpc_hook import GrpcHook
return GrpcHook(grpc_conn_id=self.conn_id)
+raise AirflowException("Hook for connection type {} not
found".format(self.conn_type))
Review comment:
Thanks, updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jmcarp commented on a change in pull request #4926: [AIRFLOW-4104] Add type annotations to common classes.
jmcarp commented on a change in pull request #4926: [AIRFLOW-4104] Add type annotations to common classes. URL: https://github.com/apache/airflow/pull/4926#discussion_r267152078 ## File path: airflow/models/__init__.py ## @@ -4290,7 +4291,13 @@ def setdefault(cls, key, default, deserialize_json=False): @classmethod @provide_session -def get(cls, key, default_var=__NO_DEFAULT_SENTINEL, deserialize_json=False, session=None): +def get( +cls, +key, # type: str +default_var=__NO_DEFAULT_SENTINEL, # type: Any +deserialize_json=False, # type: bool +session=None Review comment: Either way works. From https://mypy.readthedocs.io/en/latest/python2.html, we can annotate a function like this: ```python def hello(): # type: () -> None print 'hello' ``` In this case, the arguments and the type annotation wouldn't fit on a single line, so I thought it made sense to split the arguments up, but I'd be happy to reformat. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jmcarp commented on a change in pull request #4926: [AIRFLOW-4104] Add type annotations to common classes.
jmcarp commented on a change in pull request #4926: [AIRFLOW-4104] Add type annotations to common classes. URL: https://github.com/apache/airflow/pull/4926#discussion_r267149990 ## File path: airflow/models/__init__.py ## @@ -3051,8 +3052,8 @@ def __init__( ) self.schedule_interval = schedule_interval -if isinstance(schedule_interval, Hashable) and schedule_interval in cron_presets: -self._schedule_interval = cron_presets.get(schedule_interval) +if isinstance(schedule_interval, six.string_types) and schedule_interval in cron_presets: Review comment: It looks like the `Hashable` check was added because `relativedelta` objects aren't hashable in `python-dateutil` before version 2.7. It turns out that the keys of `cron_presets` are always strings, so we can check that `schedule_interval` is a string instead of checking that it's hashable. We could also require `python-dateutil>=2.7` and drop both checks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jmcarp commented on a change in pull request #4926: [AIRFLOW-4104] Add type annotations to common classes.
jmcarp commented on a change in pull request #4926: [AIRFLOW-4104] Add type
annotations to common classes.
URL: https://github.com/apache/airflow/pull/4926#discussion_r267148079
##
File path: airflow/models/__init__.py
##
@@ -2065,44 +2061,45 @@ class derived from this one results in the creation of
a task object,
@apply_defaults
def __init__(
-self,
-task_id,
-owner=configuration.conf.get('operators', 'DEFAULT_OWNER'),
-email=None,
-email_on_retry=True,
-email_on_failure=True,
-retries=0,
-retry_delay=timedelta(seconds=300),
-retry_exponential_backoff=False,
-max_retry_delay=None,
-start_date=None,
-end_date=None,
-schedule_interval=None, # not hooked as of now
-depends_on_past=False,
-wait_for_downstream=False,
-dag=None,
-params=None,
-default_args=None,
-adhoc=False,
-priority_weight=1,
-weight_rule=WeightRule.DOWNSTREAM,
-queue=configuration.conf.get('celery', 'default_queue'),
-pool=None,
-sla=None,
-execution_timeout=None,
-on_failure_callback=None,
-on_success_callback=None,
-on_retry_callback=None,
-trigger_rule=TriggerRule.ALL_SUCCESS,
-resources=None,
-run_as_user=None,
-task_concurrency=None,
-executor_config=None,
-do_xcom_push=True,
-inlets=None,
-outlets=None,
-*args,
-**kwargs):
+self,
+task_id, # type: str
+owner=configuration.conf.get('operators', 'DEFAULT_OWNER'), # type:
str
+email=None, # type: Optional[str]
+email_on_retry=True, # type: bool
+email_on_failure=True, # type: bool
+retries=0, # type: int
+retry_delay=timedelta(seconds=300), # type: timedelta
+retry_exponential_backoff=False, # type: bool
+max_retry_delay=None, # type: Optional[datetime]
+start_date=None, # type: Optional[datetime]
+end_date=None, # type: Optional[datetime]
+schedule_interval=None, # not hooked as of now
Review comment:
Not sure! This comment was already here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #4948: [AIRFLOW-4129] Escape HTML in generated tooltips (#4944)
codecov-io commented on issue #4948: [AIRFLOW-4129] Escape HTML in generated tooltips (#4944) URL: https://github.com/apache/airflow/pull/4948#issuecomment-474634025 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4948?src=pr=h1) Report > Merging [#4948](https://codecov.io/gh/apache/airflow/pull/4948?src=pr=desc) into [v1-10-stable](https://codecov.io/gh/apache/airflow/commit/229509939e7999fdb498e2a2d6de2509c28df358?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4948?src=pr=tree) ```diff @@ Coverage Diff @@ ## v1-10-stable#4948 +/- ## = Coverage 77.24% 77.24% = Files 219 219 Lines 1733017330 = Hits 1338613386 Misses 3944 3944 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4948?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4948?src=pr=footer). Last update [2295099...e0a5052](https://codecov.io/gh/apache/airflow/pull/4948?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on a change in pull request #4889: [AIRFLOW-4057] statsd should escape invalid characters
milton0825 commented on a change in pull request #4889: [AIRFLOW-4057] statsd
should escape invalid characters
URL: https://github.com/apache/airflow/pull/4889#discussion_r267144768
##
File path: airflow/stats.py
##
@@ -0,0 +1,124 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from functools import wraps
+import logging
+import socket
+import string
+import textwrap
+from typing import Any
+
+from airflow import configuration as conf
+from airflow.exceptions import InvalidStatsNameException
+
+log = logging.getLogger(__name__)
+
+
+class DummyStatsLogger(object):
+@classmethod
+def incr(cls, stat, count=1, rate=1):
+pass
+
+@classmethod
+def decr(cls, stat, count=1, rate=1):
+pass
+
+@classmethod
+def gauge(cls, stat, value, rate=1, delta=False):
+pass
+
+@classmethod
+def timing(cls, stat, dt):
+pass
+
+
+# Only characters in the character set are considered valid
+# for the stat_name if stat_name_default_handler is used.
+ALLOWED_CHARACTERS = set(string.ascii_letters + string.digits + '_.-')
+
+
+def stat_name_default_handler(stat_name, max_length=250):
+if not isinstance(stat_name, str):
+raise InvalidStatsNameException('The stat_name has to be a string')
+if len(stat_name) > max_length:
+raise InvalidStatsNameException(textwrap.dedent("""\
+The stat_name ({stat_name}) has to be less than {max_length}
characters.
+""".format(stat_name=stat_name, max_length=max_length)))
+if not all((c in ALLOWED_CHARACTERS) for c in stat_name):
+raise InvalidStatsNameException(textwrap.dedent("""\
+The stat name ({stat_name}) has to be composed with characters in
+{allowed_characters}.
+""".format(stat_name=stat_name,
+ allowed_characters=ALLOWED_CHARACTERS)))
+return stat_name
+
+
+def validate_stat(f):
+@wraps(f)
+def wrapper(stat, *args, **kwargs):
+try:
+from airflow.plugins_manager import stat_name_handler
+if stat_name_handler:
+handle_stat_name_func = stat_name_handler
+else:
+handle_stat_name_func = stat_name_default_handler
+stat_name = handle_stat_name_func(stat)
+except Exception as err:
+log.warning('Invalid stat name: {stat}.'.format(stat=stat), err)
+return
Review comment:
Say someone introduced a `stat_name` that is invalid, I think we should warn
them and push them to rename it instead of showing the escaped metric.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3977) Incorrect example about the interaction between skipped tasks and trigger rules in documentation.
[ https://issues.apache.org/jira/browse/AIRFLOW-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796635#comment-16796635 ] ASF subversion and git services commented on AIRFLOW-3977: -- Commit 9bdb7d6fbcabf4b90710a33019ef733b7a908612 in airflow's branch refs/heads/v1-10-stable from Chen Tong [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9bdb7d6 ] [AIRFLOW-3977] Add examples of trigger rules in doc (#4805) The current LatestOnlyOperator will skip all downstream tasks blindly. The doc shows a wrong behavior. It also shows an incorrect example about the interaction between skipped tasks and trigger rules. I replace it with another example using BranchingOperator in schedule level. This fix can resolve this ticket: https://issues.apache.org/jira/browse/AIRFLOW-3977 Also, https://issues.apache.org/jira/browse/AIRFLOW-1784 > Incorrect example about the interaction between skipped tasks and trigger > rules in documentation. > - > > Key: AIRFLOW-3977 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3977 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2 >Reporter: Chen Tong >Assignee: Chen Tong >Priority: Major > Labels: documentaion > Fix For: 1.10.3 > > > Current LatestOnlyOperator will skip all downstream tasks blindly. > BranchingOperator could be a better example to show how trigger rules > interacted with skipped tasks in schedule level. > This fix can also resolve this ticket: > https://issues.apache.org/jira/browse/AIRFLOW-1784 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] milton0825 commented on a change in pull request #4889: [AIRFLOW-4057] statsd should escape invalid characters
milton0825 commented on a change in pull request #4889: [AIRFLOW-4057] statsd
should escape invalid characters
URL: https://github.com/apache/airflow/pull/4889#discussion_r267142933
##
File path: airflow/stats.py
##
@@ -0,0 +1,124 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from functools import wraps
+import logging
+import socket
+import string
+import textwrap
+from typing import Any
+
+from airflow import configuration as conf
+from airflow.exceptions import InvalidStatsNameException
+
+log = logging.getLogger(__name__)
+
+
+class DummyStatsLogger(object):
+@classmethod
+def incr(cls, stat, count=1, rate=1):
+pass
+
+@classmethod
+def decr(cls, stat, count=1, rate=1):
+pass
+
+@classmethod
+def gauge(cls, stat, value, rate=1, delta=False):
+pass
+
+@classmethod
+def timing(cls, stat, dt):
+pass
+
+
+# Only characters in the character set are considered valid
+# for the stat_name if stat_name_default_handler is used.
+ALLOWED_CHARACTERS = set(string.ascii_letters + string.digits + '_.-')
+
+
+def stat_name_default_handler(stat_name, max_length=250):
+if not isinstance(stat_name, str):
+raise InvalidStatsNameException('The stat_name has to be a string')
+if len(stat_name) > max_length:
+raise InvalidStatsNameException(textwrap.dedent("""\
+The stat_name ({stat_name}) has to be less than {max_length}
characters.
+""".format(stat_name=stat_name, max_length=max_length)))
+if not all((c in ALLOWED_CHARACTERS) for c in stat_name):
+raise InvalidStatsNameException(textwrap.dedent("""\
+The stat name ({stat_name}) has to be composed with characters in
+{allowed_characters}.
+""".format(stat_name=stat_name,
+ allowed_characters=ALLOWED_CHARACTERS)))
+return stat_name
+
+
+def validate_stat(f):
+@wraps(f)
+def wrapper(stat, *args, **kwargs):
+try:
+from airflow.plugins_manager import stat_name_handler
+if stat_name_handler:
+handle_stat_name_func = stat_name_handler
+else:
+handle_stat_name_func = stat_name_default_handler
+stat_name = handle_stat_name_func(stat)
+except Exception as err:
+log.warning('Invalid stat name: {stat}.'.format(stat=stat), err)
+return
Review comment:
I am proposing that we log warning instead of escaping as our default
behavior.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3875) simplify SlackWebhookHook code and change docstring
[ https://issues.apache.org/jira/browse/AIRFLOW-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796630#comment-16796630 ] ASF subversion and git services commented on AIRFLOW-3875: -- Commit 288c9a509d7d2aeb2fb5c21b6d5037feaa012eb2 in airflow's branch refs/heads/v1-10-stable from zhongjiajie [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=288c9a5 ] [AIRFLOW-3875] Simplify SlackWebhookHook code and change docstring (#4696) > simplify SlackWebhookHook code and change docstring > --- > > Key: AIRFLOW-3875 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3875 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Minor > Labels: easyfix > Fix For: 1.10.3 > > > SlackWebhookHook __init__ function do not need self.http_conn_id, and > docstring missing some necessary str -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3636) Fix a test introduced in PR #4425
[ https://issues.apache.org/jira/browse/AIRFLOW-3636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796638#comment-16796638 ] ASF subversion and git services commented on AIRFLOW-3636: -- Commit dc8385b47ffe72569ac4515cfa677829a3a3da20 in airflow's branch refs/heads/v1-10-stable from Ping Zhang [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=dc8385b ] [AIRFLOW-3636] Fix a test introduced in #4425 (#4446) > Fix a test introduced in PR #4425 > - > > Key: AIRFLOW-3636 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3636 > Project: Apache Airflow > Issue Type: Bug >Reporter: Ping Zhang >Assignee: Ping Zhang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3881) Correct hive_hook to_csv row number
[ https://issues.apache.org/jira/browse/AIRFLOW-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796631#comment-16796631 ] ASF subversion and git services commented on AIRFLOW-3881: -- Commit 037b6d67c494d2a18de86be48d4ff137c5e304bf in airflow's branch refs/heads/v1-10-stable from zhongjiajie [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=037b6d6 ] [AIRFLOW-3881] Correct to_csv row number (#4699) > Correct hive_hook to_csv row number > --- > > Key: AIRFLOW-3881 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3881 > Project: Apache Airflow > Issue Type: Bug > Components: hive_hooks >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Minor > Labels: easyfix > Fix For: 2.0.0, 1.10.3 > > > hive_hooks to_csv function had wrong row number each path so far, this Jira > ticket is correct row number -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2511) Subdag failed by scheduler deadlock
[
https://issues.apache.org/jira/browse/AIRFLOW-2511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796633#comment-16796633
]
ASF subversion and git services commented on AIRFLOW-2511:
--
Commit f559d43abf4e53a3a7afbfba4b3469ffddf9182c in airflow's branch
refs/heads/v1-10-stable from fenglu-g
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f559d43 ]
[AIRFLOW-2511] Fix improper failed session commit handling causing deadlocks
(#4769)
> Subdag failed by scheduler deadlock
> ---
>
> Key: AIRFLOW-2511
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2511
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Yohei Onishi
>Assignee: Feng Lu
>Priority: Major
> Fix For: 1.10.3
>
>
> I am using subdag and sometimes main dag marked failed because of the
> following error. In this case, tasks in the subdag stopped.
> {code:java}
> hourly_dag = DAG(
> hourly_dag_name,
> default_args=dag_default_args,
> params=dag_custom_params,
> schedule_interval=config_values.hourly_job_interval,
> max_active_runs=2)
> hourly_subdag = SubDagOperator(
> task_id='s3_to_hive',
> subdag=LoadFromS3ToHive(
> hourly_dag,
> 's3_to_hive'),
> dag=hourly_dag)
> {code}
> I got this error in main dag. bug in scheduler?
> {code:java}
> [2018-05-22 21:52:19,683] {models.py:1595} ERROR - This Session's transaction
> has been rolled back due to a previous exception during flush. To begin a new
> transaction with this Session, first issue Session.rollback(). Original
> exception was: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found
> when trying to get lock; try restarting transaction') [SQL: 'UPDATE
> task_instance SET state=%s WHERE task_instance.task_id = %s AND
> task_instance.dag_id = %s AND task_instance.execution_date = %s']
> [parameters: ('queued', 'transfer_from_tmp_table_into_cleaned_table',
> 'rfid_warehouse_carton_wh_g_dl_dwh_csv_uqjp_1h.s3_to_hive',
> datetime.datetime(2018, 5, 7, 5, 2))] (Background on this error at:
> http://sqlalche.me/e/e3q8)
> Traceback (most recent call last):
> sqlalchemy.exc.InvalidRequestError: This Session's transaction has been
> rolled back due to a previous exception during flush. To begin a new
> transaction with this Session, first issue Session.rollback(). Original
> exception was: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found
> when trying to get lock; try restarting transaction') [SQL: 'UPDATE
> task_instance SET state=%s WHERE task_instance.task_id = %s AND
> task_instance.dag_id = %s AND task_instance.execution_date = %s']
> [parameters: ('queued', 'transfer_from_tmp_table_into_cleaned_table',
> 'rfid_warehouse_carton_wh_g_dl_dwh_csv_uqjp_1h.s3_to_hive',
> datetime.datetime(2018, 5, 7, 5, 2))] (Background on this error at:
> http://sqlalche.me/e/e3q8)
> [2018-05-22 21:52:19,687] {models.py:1624} INFO - Marking task as FAILED.
> [2018-05-22 21:52:19,688] {base_task_runner.py:98} INFO - Subtask:
> [2018-05-22 21:52:19,688] {slack_hook.py:143} INFO - Message is prepared:
> [2018-05-22 21:52:19,688] {base_task_runner.py:98} INFO - Subtask:
> {"attachments": [{"color": "danger", "text": "", "fields": [{"title": "DAG",
> "value":
> "",
> "short": true}, {"title": "Owner", "value": "airflow", "short": true},
> {"title": "Task", "value": "s3_to_hive", "short": false}, {"title": "Status",
> "value": "FAILED", "short": false}, {"title": "Execution Time", "value":
> "2018-05-07T05:02:00", "short": true}, {"title": "Duration", "value":
> "826.305929", "short": true}, {"value":
> " Task Log>", "short": false}]}]}
> [2018-05-22 21:52:19,688] {models.py:1638} ERROR - Failed at executing
> callback
> [2018-05-22 21:52:19,688] {models.py:1639} ERROR - This Session's transaction
> has been rolled back due to a previous exception during flush. To begin a new
> transaction with this Session, first issue Session.rollback(). Original
> exception was: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found
> when trying to get lock; try restarting transaction') [SQL: 'UPDATE
> task_instance SET state=%s WHERE task_instance.task_id = %s AND
> task_instance.dag_id = %s AND task_instance.execution_date = %s']
> [parameters: ('queued', 'transfer_from_tmp_table_into_cleaned_table',
> 'rfid_warehouse_carton_wh_g_dl_dwh_csv_uqjp_1h.s3_to_hive',
> datetime.datetime(2018, 5, 7, 5, 2))] (Background on this error at:
>
[jira] [Commented] (AIRFLOW-3734) function `load_file` will not run if param `partition` is None
[
https://issues.apache.org/jira/browse/AIRFLOW-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796628#comment-16796628
]
ASF subversion and git services commented on AIRFLOW-3734:
--
Commit e9a1bf8b6e85b19f38c6fd4e0e661e9b69cb7d70 in airflow's branch
refs/heads/v1-10-stable from zhongjiajie
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e9a1bf8 ]
[AIRFLOW-3734] Fix hql not run when partition is None (#4561)
> function `load_file` will not run if param `partition` is None
> --
>
> Key: AIRFLOW-3734
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3734
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks
>Affects Versions: 1.10.1
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Labels: easyfix, patch
> Fix For: 1.10.3
>
>
> the code in master branch in hive_hooks.py in function load_file like below
> {code:java}
> hql = "LOAD DATA LOCAL INPATH '{filepath}' "
> if overwrite:
> hql += "OVERWRITE "
> hql += "INTO TABLE {table} "
> if partition:
> pvals = ", ".join(
> ["{0}='{1}'".format(k, v) for k, v in partition.items()])
> hql += "PARTITION ({pvals});"
> # As a workaround for HIVE-10541, add a newline character
> # at the end of hql (AIRFLOW-2412).
> hql += '\n'{code}
> when param partition is None, the hql will like this
> {code:java}
> LOAD DATA LOCAL INPATH OVERWRITE INTO TABLE \n{code}
> without `;` syntax the hql will not run, so I think we should move `;` from
> `hql += "PARTITION (\{pvals});"` to `hql += ';\n'`
>
> so the code will like
> {code:java}
> hql = "LOAD DATA LOCAL INPATH '{filepath}' "
> if overwrite:
> hql += "OVERWRITE "
> hql += "INTO TABLE {table} "
> if partition:
> pvals = ", ".join(
> ["{0}='{1}'".format(k, v) for k, v in partition.items()])
> hql += "PARTITION ({pvals})"
> # As a workaround for HIVE-10541, add a newline character
> # at the end of hql (AIRFLOW-2412).
> hql += ';\n'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1784) SKIPPED status is being cascading wrongly
[ https://issues.apache.org/jira/browse/AIRFLOW-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796636#comment-16796636 ] ASF subversion and git services commented on AIRFLOW-1784: -- Commit 9bdb7d6fbcabf4b90710a33019ef733b7a908612 in airflow's branch refs/heads/v1-10-stable from Chen Tong [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9bdb7d6 ] [AIRFLOW-3977] Add examples of trigger rules in doc (#4805) The current LatestOnlyOperator will skip all downstream tasks blindly. The doc shows a wrong behavior. It also shows an incorrect example about the interaction between skipped tasks and trigger rules. I replace it with another example using BranchingOperator in schedule level. This fix can resolve this ticket: https://issues.apache.org/jira/browse/AIRFLOW-3977 Also, https://issues.apache.org/jira/browse/AIRFLOW-1784 > SKIPPED status is being cascading wrongly > - > > Key: AIRFLOW-1784 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1784 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.8.2 > Environment: Ubuntu 16.04.3 LTS > Python 2.7.12 > CeleryExecutor: 2-nodes cluster >Reporter: Dmytro Kulyk >Priority: Critical > Labels: documentation, latestonly, operators > Attachments: Capture_graph.JPG, Capture_tree2.JPG, cube_update.py > > > After implementation of AIRFLOW-1296 within 1.8.2 there is an wrong behavior > of LatestOnlyOperator which is forcing SKIPPED status cascading despite of > TriggerRule='all_done' set > Which is opposite to documented > [here|https://airflow.incubator.apache.org/concepts.html#latest-run-only] > *Expected Behavior:* > dummy task and all downstreams (update_*) should not be skipped > Full listings are attached > 1.8.1 did not have such issue -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3623) Support download log file from UI
[ https://issues.apache.org/jira/browse/AIRFLOW-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796637#comment-16796637 ] ASF subversion and git services commented on AIRFLOW-3623: -- Commit 4579a007ef10280c05cc6d33ce0eed0fb8b50508 in airflow's branch refs/heads/v1-10-stable from Ping Zhang [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4579a00 ] [AIRFLOW-3623] Support download logs by attempts from UI (#4425) > Support download log file from UI > - > > Key: AIRFLOW-3623 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3623 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Ping Zhang >Assignee: Ping Zhang >Priority: Major > Labels: newbie > Fix For: 2.0.0 > > > for some large log files, it is not a good idea to fetch and render in the > UI. Adding the ability to let users to download by try_number in the dag > modal. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3962) Scheduler throwing an exception while creating dag_run for a dag which doesn't contain any task
[
https://issues.apache.org/jira/browse/AIRFLOW-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796632#comment-16796632
]
ASF subversion and git services commented on AIRFLOW-3962:
--
Commit 0196b5f05f1032621024389b6a089aedc3921b69 in airflow's branch
refs/heads/v1-10-stable from Vardan Gupta
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=0196b5f ]
[AIRFLOW-3962] Added graceful handling for creation of dag_run of a dag which
doesn't have any task (#4781)
> Scheduler throwing an exception while creating dag_run for a dag which
> doesn't contain any task
> ---
>
> Key: AIRFLOW-3962
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3962
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.2
>Reporter: Vardan Gupta
>Assignee: Vardan Gupta
>Priority: Major
> Fix For: 1.10.3
>
>
> Steps to Reproduce
> * Copy a python dag file which doesn't contain in tasks(Sample python
> code available at the bottom)
> * Make sure dag is not paused
> * During create_dag_run method call of schedule the time of file processing,
> task_start_dates list will come out to be empty
> [here|https://github.com/apache/airflow/blob/f3dfc9495be41221bd08ea753555df0975deb961/airflow/jobs.py#L840],
> due to this *_next_run_date_* will remain None
> * Above will cause *_next_run_date > timezone.utcnow()_* comparison
> [here|https://github.com/apache/airflow/blob/f3dfc9495be41221bd08ea753555df0975deb961/airflow/jobs.py#L869]
> to fail and resulting in below exception
> {code:java}
> Traceback (most recent call last):
> File "/usr/src/venv/local/lib/python2.7/site-packages/airflow/jobs.py", line
> 398, in helper
> pickle_dags)
> File "/usr/src/venv/local/lib/python2.7/site-packages/airflow/utils/db.py",
> line 69, in wrapper
> result = func(*args, **kwargs)
> File "/usr/src/venv/local/lib/python2.7/site-packages/airflow/jobs.py", line
> 1749, in process_file
> self._process_dags(dagbag, dags, ti_keys_to_schedule)
> File "/usr/src/venv/local/lib/python2.7/site-packages/airflow/jobs.py", line
> 1438, in _process_dags
> dag_run = self.create_dag_run(dag)
> File "/usr/src/venv/local/lib/python2.7/site-packages/airflow/utils/db.py",
> line 69, in wrapper
> result = func(*args, **kwargs)
> File "/usr/src/venv/local/lib/python2.7/site-packages/airflow/jobs.py", line
> 869, in create_dag_run
> if next_run_date > timezone.utcnow():
> TypeError: can't compare datetime.datetime to NoneType{code}
>
> Sample Python file without any task
>
> {code:java}
> import airflow
> from airflow import DAG
> from airflow.utils.trigger_rule import TriggerRule
> from datetime import datetime, timedelta
> import dateutil.parser
> # Add task level imports
>
> default_global_args = {
> 'owner': 'owner1',
> 'email': ['no-re...@domain.com'],
> 'email_on_failure': False,
> 'email_on_retry': False,
> 'start_date': dateutil.parser.parse('2019-02-21T00:00:00.000+05:30',
> ignoretz=True),
> 'end_date': dateutil.parser.parse('2019-02-22T00:00:00.000+05:30',
> ignoretz=True)
> }
> dag = DAG(
> dag_id = 'edce5a655a5e73ff0a494113_mlexperiment03_34sik4',
> default_args = default_global_args,
> schedule_interval = '@once',
> )
> {code}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3733) raise NameError if hql result had no record
[ https://issues.apache.org/jira/browse/AIRFLOW-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796629#comment-16796629 ] ASF subversion and git services commented on AIRFLOW-3733: -- Commit ee688f6987221788c1c67cc15b6b49938f8d61cb in airflow's branch refs/heads/v1-10-stable from zhongjiajie [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ee688f6 ] [AIRFLOW-3733] Don't raise NameError in HQL hook to_csv when no rows returned (#4560) > raise NameError if hql result had no record > --- > > Key: AIRFLOW-3733 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3733 > Project: Apache Airflow > Issue Type: Bug > Components: hive_hooks >Affects Versions: 1.10.1 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: easyfix > Fix For: 1.10.3 > > > [to_csv|https://github.com/apache/airflow/blob/233f6c750c543a669c5319c7167a68a1f5b33c16/airflow/hooks/hive_hooks.py#L881-L903] > function will raise `NameError` while hql result had not record, we should > init variable i like > [oracle_hooks|https://github.com/apache/airflow/blob/233f6c750c543a669c5319c7167a68a1f5b33c16/airflow/hooks/oracle_hook.py#L92-L124] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2715) Dataflow template operator dosenot support region parameter
[ https://issues.apache.org/jira/browse/AIRFLOW-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796627#comment-16796627 ] ASF subversion and git services commented on AIRFLOW-2715: -- Commit f00ce81a56d1c0876ff75a4464e9dfd27aeed408 in airflow's branch refs/heads/v1-10-stable from Jan Hicken [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f00ce81 ] [AIRFLOW-2715] Use region setting when launching Dataflow templates (#4139) To launch an instance of a Dataflow template in the configured region, the API service.projects().locations().teplates() instead of service.projects().templates() has to be used. Otherwise, all jobs will always be started in us-central1. In case there is no region configured, the default region `us-central1` will get picked up. To make it even worse, the polling for the job status already honors the region parameter and will search for the job in the wrong region in the current implementation. Because the job's status is not found, the corresponding Airflow task will hang. > Dataflow template operator dosenot support region parameter > --- > > Key: AIRFLOW-2715 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2715 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.9.0 >Reporter: Mohammed Tameem >Priority: Critical > Fix For: 1.10.3 > > > The DataflowTemplateOperator uses dataflow.projects.templates.launch which > has a region parameter but only supports execution of the dataflow job in the > us-central1 region. Alternatively there is another api, > dataflow.projects.locations.templates.launch which supports execution of the > template in all regional endpoints provided by google cloud. > It would be great if, > # The base REST API of this operator could be changed from > "dataflow.projects.templates.launch" to > "dataflow.projects.locations.templates.launch" > # A templated region paramter was included in the operator to run the > dataflow job in the requested regional endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3977) Incorrect example about the interaction between skipped tasks and trigger rules in documentation.
[ https://issues.apache.org/jira/browse/AIRFLOW-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796634#comment-16796634 ] ASF subversion and git services commented on AIRFLOW-3977: -- Commit 9bdb7d6fbcabf4b90710a33019ef733b7a908612 in airflow's branch refs/heads/v1-10-stable from Chen Tong [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9bdb7d6 ] [AIRFLOW-3977] Add examples of trigger rules in doc (#4805) The current LatestOnlyOperator will skip all downstream tasks blindly. The doc shows a wrong behavior. It also shows an incorrect example about the interaction between skipped tasks and trigger rules. I replace it with another example using BranchingOperator in schedule level. This fix can resolve this ticket: https://issues.apache.org/jira/browse/AIRFLOW-3977 Also, https://issues.apache.org/jira/browse/AIRFLOW-1784 > Incorrect example about the interaction between skipped tasks and trigger > rules in documentation. > - > > Key: AIRFLOW-3977 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3977 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2 >Reporter: Chen Tong >Assignee: Chen Tong >Priority: Major > Labels: documentaion > Fix For: 1.10.3 > > > Current LatestOnlyOperator will skip all downstream tasks blindly. > BranchingOperator could be a better example to show how trigger rules > interacted with skipped tasks in schedule level. > This fix can also resolve this ticket: > https://issues.apache.org/jira/browse/AIRFLOW-1784 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4070) AirflowException on repeated dependencies prevents automated dependencies
[ https://issues.apache.org/jira/browse/AIRFLOW-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-4070: --- Fix Version/s: (was: 2.0.0) 1.10.3 > AirflowException on repeated dependencies prevents automated dependencies > - > > Key: AIRFLOW-4070 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4070 > Project: Apache Airflow > Issue Type: Bug >Reporter: Newton Le >Assignee: Newton Le >Priority: Major > Fix For: 1.10.3 > > > We are trying to build automated dependencies based on data. For example, if > Task A requires data from Task B, then a task dependency will be > automatically generated. At the same time, we still want to allow users to > specify explicit dependencies, e.g. `task_a >> task_b`. > Airflow's current way of raising an exception on duplicate dependencies > currently blocks this implementation. Also, I can't think of a good > justification for it. For the purposes I can imagine, logging a warning seems > more than sufficient. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4124) Add get table and get location of table to AWS Glue Hook
[
https://issues.apache.org/jira/browse/AIRFLOW-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor resolved AIRFLOW-4124.
Resolution: Fixed
> Add get table and get location of table to AWS Glue Hook
>
>
> Key: AIRFLOW-4124
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4124
> Project: Apache Airflow
> Issue Type: New Feature
> Components: aws
>Reporter: Bryan Yang
>Assignee: Bryan Yang
>Priority: Minor
> Fix For: 1.10.3
>
>
> There are many functions in AWS Glue API. {{get_table}} is one of the
> important function of them.
> For the convenience reason, I added {{get_location}} function to extract the
> location of the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4124) Add get table and get location of table to AWS Glue Hook
[
https://issues.apache.org/jira/browse/AIRFLOW-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796624#comment-16796624
]
ASF subversion and git services commented on AIRFLOW-4124:
--
Commit afb108d3b7658b71c1b27bdf7e6c6e2dd1962e35 in airflow's branch
refs/heads/master from Bryan Yang
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=afb108d ]
[AIRFLOW-4124] add get_table and get_table_location in aws_glue_hook and tests
(#4942)
> Add get table and get location of table to AWS Glue Hook
>
>
> Key: AIRFLOW-4124
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4124
> Project: Apache Airflow
> Issue Type: New Feature
> Components: aws
>Reporter: Bryan Yang
>Assignee: Bryan Yang
>Priority: Minor
> Fix For: 1.10.3
>
>
> There are many functions in AWS Glue API. {{get_table}} is one of the
> important function of them.
> For the convenience reason, I added {{get_location}} function to extract the
> location of the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4124) Add get table and get location of table to AWS Glue Hook
[
https://issues.apache.org/jira/browse/AIRFLOW-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796623#comment-16796623
]
ASF GitHub Bot commented on AIRFLOW-4124:
-
ashb commented on pull request #4942: [AIRFLOW-4124] add get_table and
get_table_location in aws_glue_hook and tests
URL: https://github.com/apache/airflow/pull/4942
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add get table and get location of table to AWS Glue Hook
>
>
> Key: AIRFLOW-4124
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4124
> Project: Apache Airflow
> Issue Type: New Feature
> Components: aws
>Reporter: Bryan Yang
>Assignee: Bryan Yang
>Priority: Minor
> Fix For: 1.10.3
>
>
> There are many functions in AWS Glue API. {{get_table}} is one of the
> important function of them.
> For the convenience reason, I added {{get_location}} function to extract the
> location of the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4124) Add get table and get location of table to AWS Glue Hook
[
https://issues.apache.org/jira/browse/AIRFLOW-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-4124:
---
Fix Version/s: 1.10.3
> Add get table and get location of table to AWS Glue Hook
>
>
> Key: AIRFLOW-4124
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4124
> Project: Apache Airflow
> Issue Type: New Feature
> Components: aws
>Reporter: Bryan Yang
>Assignee: Bryan Yang
>Priority: Minor
> Fix For: 1.10.3
>
>
> There are many functions in AWS Glue API. {{get_table}} is one of the
> important function of them.
> For the convenience reason, I added {{get_location}} function to extract the
> location of the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] ashb merged pull request #4942: [AIRFLOW-4124] add get_table and get_table_location in aws_glue_hook and tests
ashb merged pull request #4942: [AIRFLOW-4124] add get_table and get_table_location in aws_glue_hook and tests URL: https://github.com/apache/airflow/pull/4942 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #4889: [AIRFLOW-4057] statsd should escape invalid characters
ashb commented on a change in pull request #4889: [AIRFLOW-4057] statsd should
escape invalid characters
URL: https://github.com/apache/airflow/pull/4889#discussion_r267139899
##
File path: airflow/stats.py
##
@@ -0,0 +1,124 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from functools import wraps
+import logging
+import socket
+import string
+import textwrap
+from typing import Any
+
+from airflow import configuration as conf
+from airflow.exceptions import InvalidStatsNameException
+
+log = logging.getLogger(__name__)
+
+
+class DummyStatsLogger(object):
+@classmethod
+def incr(cls, stat, count=1, rate=1):
+pass
+
+@classmethod
+def decr(cls, stat, count=1, rate=1):
+pass
+
+@classmethod
+def gauge(cls, stat, value, rate=1, delta=False):
+pass
+
+@classmethod
+def timing(cls, stat, dt):
+pass
+
+
+# Only characters in the character set are considered valid
+# for the stat_name if stat_name_default_handler is used.
+ALLOWED_CHARACTERS = set(string.ascii_letters + string.digits + '_.-')
+
+
+def stat_name_default_handler(stat_name, max_length=250):
+if not isinstance(stat_name, str):
+raise InvalidStatsNameException('The stat_name has to be a string')
+if len(stat_name) > max_length:
+raise InvalidStatsNameException(textwrap.dedent("""\
+The stat_name ({stat_name}) has to be less than {max_length}
characters.
+""".format(stat_name=stat_name, max_length=max_length)))
+if not all((c in ALLOWED_CHARACTERS) for c in stat_name):
+raise InvalidStatsNameException(textwrap.dedent("""\
+The stat name ({stat_name}) has to be composed with characters in
+{allowed_characters}.
+""".format(stat_name=stat_name,
+ allowed_characters=ALLOWED_CHARACTERS)))
+return stat_name
+
+
+def validate_stat(f):
+@wraps(f)
+def wrapper(stat, *args, **kwargs):
+try:
+from airflow.plugins_manager import stat_name_handler
+if stat_name_handler:
+handle_stat_name_func = stat_name_handler
+else:
+handle_stat_name_func = stat_name_default_handler
+stat_name = handle_stat_name_func(stat)
+except Exception as err:
+log.warning('Invalid stat name: {stat}.'.format(stat=stat), err)
+return
Review comment:
> Users can define their own
Yes, but if they are using statsd, all statsd have the same character rules,
so why don't we do that work once, in Airflow?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mans2singh commented on issue #4887: [AIRFLOW-4055] Add AWS SQS Sensor
mans2singh commented on issue #4887: [AIRFLOW-4055] Add AWS SQS Sensor URL: https://github.com/apache/airflow/pull/4887#issuecomment-474626878 @zhongjiajie - I've updated the code based on your comments. Please let me know if you have additional advice. Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 …
ashb commented on issue #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 … URL: https://github.com/apache/airflow/pull/4941#issuecomment-474626813 @dimberman Not sure if you have perms in Jira yet, but when merging a PR please also mark the issue as Resolved (not Closed) and set the "Fix Version" - either 2.0.0 if it's a big change/against new code in master, or 1.10.3 (currently) if you want the fix to be cherry-picked (by me usually) in to the release. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #4926: [AIRFLOW-4104] Add type annotations to common classes.
ashb commented on a change in pull request #4926: [AIRFLOW-4104] Add type annotations to common classes. URL: https://github.com/apache/airflow/pull/4926#discussion_r267138726 ## File path: airflow/models/__init__.py ## @@ -3051,8 +3052,8 @@ def __init__( ) self.schedule_interval = schedule_interval -if isinstance(schedule_interval, Hashable) and schedule_interval in cron_presets: -self._schedule_interval = cron_presets.get(schedule_interval) +if isinstance(schedule_interval, six.string_types) and schedule_interval in cron_presets: Review comment: This looks like a possible code change. Please double check with the PR that introduced the "Hashable" here for the reasoning This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #4926: [AIRFLOW-4104] Add type annotations to common classes.
ashb commented on a change in pull request #4926: [AIRFLOW-4104] Add type
annotations to common classes.
URL: https://github.com/apache/airflow/pull/4926#discussion_r267139140
##
File path: airflow/models/connection.py
##
@@ -251,6 +251,7 @@ def get_hook(self):
elif self.conn_type == 'grpc':
from airflow.contrib.hooks.grpc_hook import GrpcHook
return GrpcHook(grpc_conn_id=self.conn_id)
+raise AirflowException("Hook for connection type {} not
found".format(self.conn_type))
Review comment:
"Not found" isn't quite right - unknown hook type would be closer I think.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #4926: [AIRFLOW-4104] Add type annotations to common classes.
ashb commented on a change in pull request #4926: [AIRFLOW-4104] Add type annotations to common classes. URL: https://github.com/apache/airflow/pull/4926#discussion_r267139037 ## File path: airflow/models/__init__.py ## @@ -4290,7 +4291,13 @@ def setdefault(cls, key, default, deserialize_json=False): @classmethod @provide_session -def get(cls, key, default_var=__NO_DEFAULT_SENTINEL, deserialize_json=False, session=None): +def get( +cls, +key, # type: str +default_var=__NO_DEFAULT_SENTINEL, # type: Any +deserialize_json=False, # type: bool +session=None Review comment: General style point: On these "shorter" fns is it possible to put the type annotation at the end instead of inline? Or is this the more commonly accepted style for type comments on func args? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #4926: [AIRFLOW-4104] Add type annotations to common classes.
ashb commented on a change in pull request #4926: [AIRFLOW-4104] Add type
annotations to common classes.
URL: https://github.com/apache/airflow/pull/4926#discussion_r267138457
##
File path: airflow/models/__init__.py
##
@@ -2065,44 +2061,45 @@ class derived from this one results in the creation of
a task object,
@apply_defaults
def __init__(
-self,
-task_id,
-owner=configuration.conf.get('operators', 'DEFAULT_OWNER'),
-email=None,
-email_on_retry=True,
-email_on_failure=True,
-retries=0,
-retry_delay=timedelta(seconds=300),
-retry_exponential_backoff=False,
-max_retry_delay=None,
-start_date=None,
-end_date=None,
-schedule_interval=None, # not hooked as of now
-depends_on_past=False,
-wait_for_downstream=False,
-dag=None,
-params=None,
-default_args=None,
-adhoc=False,
-priority_weight=1,
-weight_rule=WeightRule.DOWNSTREAM,
-queue=configuration.conf.get('celery', 'default_queue'),
-pool=None,
-sla=None,
-execution_timeout=None,
-on_failure_callback=None,
-on_success_callback=None,
-on_retry_callback=None,
-trigger_rule=TriggerRule.ALL_SUCCESS,
-resources=None,
-run_as_user=None,
-task_concurrency=None,
-executor_config=None,
-do_xcom_push=True,
-inlets=None,
-outlets=None,
-*args,
-**kwargs):
+self,
+task_id, # type: str
+owner=configuration.conf.get('operators', 'DEFAULT_OWNER'), # type:
str
+email=None, # type: Optional[str]
+email_on_retry=True, # type: bool
+email_on_failure=True, # type: bool
+retries=0, # type: int
+retry_delay=timedelta(seconds=300), # type: timedelta
+retry_exponential_backoff=False, # type: bool
+max_retry_delay=None, # type: Optional[datetime]
+start_date=None, # type: Optional[datetime]
+end_date=None, # type: Optional[datetime]
+schedule_interval=None, # not hooked as of now
Review comment:
?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #4898: [AIRFLOW-XXXX] how to setup simple test env with mysql
milton0825 commented on issue #4898: [AIRFLOW-] how to setup simple test env with mysql URL: https://github.com/apache/airflow/pull/4898#issuecomment-474625399 PTAL @mik-laj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #4944: [AIRFLOW-4129] Escape HTML in generated tooltips
codecov-io commented on issue #4944: [AIRFLOW-4129] Escape HTML in generated tooltips URL: https://github.com/apache/airflow/pull/4944#issuecomment-474624216 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4944?src=pr=h1) Report > Merging [#4944](https://codecov.io/gh/apache/airflow/pull/4944?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a8f037e6de6bb30ac00383a426012b077858c3a1?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4944?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4944 +/- ## == + Coverage 75.58% 75.58% +<.01% == Files 454 454 Lines 2919729197 == + Hits2206922070 +1 + Misses 7128 7127 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4944?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4944/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `92.89% <0%> (+0.05%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4944?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4944?src=pr=footer). Last update [a8f037e...280dcf2](https://codecov.io/gh/apache/airflow/pull/4944?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jmcarp commented on issue #4926: [AIRFLOW-4104] Add type annotations to common classes.
jmcarp commented on issue #4926: [AIRFLOW-4104] Add type annotations to common classes. URL: https://github.com/apache/airflow/pull/4926#issuecomment-474623618 WDYT @ashb @feng-tao @Fokko? This makes almost no changes to the code--it's almost all annotations. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3706) Tooltip Display Issue for a Task Instance
[
https://issues.apache.org/jira/browse/AIRFLOW-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796611#comment-16796611
]
ASF GitHub Bot commented on AIRFLOW-3706:
-
ashb commented on pull request #4947: [AIRFLOW-3706] Fix tooltip max-width by
correcting ordering of CSS files
URL: https://github.com/apache/airflow/pull/4947
Make sure you have checked _all_ steps below.
### Jira
https://issues.apache.org/jira/browse/AIRFLOW-3706
### Description
The "airflowDefaultTheme.css" file is created by webpack out of
bootstrap-theme.css, which is a "base" file. But this was loaded _after_
Airflow's main.css, meaning the `max-width` we set on `.tooltip-inner`
was being replaced by the default from bootstrap.
/cc @verdan
### Tests
**Before**:
**After**:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Tooltip Display Issue for a Task Instance
> -
>
> Key: AIRFLOW-3706
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3706
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Affects Versions: 1.10.2
>Reporter: Felix Uellendall
>Assignee: Ash Berlin-Taylor
>Priority: Blocker
> Fix For: 1.10.3
>
> Attachments: 1.10.1-firefox.png, 1.10.1-safari.png,
> 1.10.2b2-firefox.png, 1.10.2b2-safari.png
>
>
> I am facing a tooltip display issue in 1.10.2b2 when hovering over a task
> instance the tooltip rectangle that is being displayed is not automatically
> resizing - for 1.10.1 it does.
> I am using the RBAC UI.
> I tested it with Firefox 64.0.2 (64-bit) and Safari Version 11.1.2
> (13605.3.8) on mac OS High Sierra Version 10.13.6
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4129) Escape HTML in tooltips
[
https://issues.apache.org/jira/browse/AIRFLOW-4129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796616#comment-16796616
]
ASF GitHub Bot commented on AIRFLOW-4129:
-
ashb commented on pull request #4948: [AIRFLOW-4129] Escape HTML in generated
tooltips (#4944)
URL: https://github.com/apache/airflow/pull/4948
Make sure you have checked _all_ steps below.
### Jira
https://issues.apache.org/jira/browse/AIRFLOW-4129
### Description
Escape anything that we expect to be plain text when building tooltips.
Many of these properties aren't user generated, but it is better to be
safe.
This is a back-port of #4944 to 1.10 branch on the RBAC and classic UI,
both of which are slightly different code layout to current master
Many of these properties aren't user generated, but it is better to be
safe
### Tests
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Escape HTML in tooltips
> ---
>
> Key: AIRFLOW-4129
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4129
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.2
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] ashb commented on issue #4944: [AIRFLOW-4129] Escape HTML in generated tooltips
ashb commented on issue #4944: [AIRFLOW-4129] Escape HTML in generated tooltips URL: https://github.com/apache/airflow/pull/4944#issuecomment-474623766 I have also created a second PR targeting the v1-10-stable branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #4948: [AIRFLOW-4129] Escape HTML in generated tooltips (#4944)
ashb commented on issue #4948: [AIRFLOW-4129] Escape HTML in generated tooltips (#4944) URL: https://github.com/apache/airflow/pull/4948#issuecomment-474623830 Please approve (if you are happy with this) but don't merge please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb opened a new pull request #4948: [AIRFLOW-4129] Escape HTML in generated tooltips (#4944)
ashb opened a new pull request #4948: [AIRFLOW-4129] Escape HTML in generated
tooltips (#4944)
URL: https://github.com/apache/airflow/pull/4948
Make sure you have checked _all_ steps below.
### Jira
https://issues.apache.org/jira/browse/AIRFLOW-4129
### Description
Escape anything that we expect to be plain text when building tooltips.
Many of these properties aren't user generated, but it is better to be
safe.
This is a back-port of #4944 to 1.10 branch on the RBAC and classic UI,
both of which are slightly different code layout to current master
Many of these properties aren't user generated, but it is better to be
safe
### Tests
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb opened a new pull request #4947: [AIRFLOW-3706] Fix tooltip max-width by correcting ordering of CSS files
ashb opened a new pull request #4947: [AIRFLOW-3706] Fix tooltip max-width by
correcting ordering of CSS files
URL: https://github.com/apache/airflow/pull/4947
Make sure you have checked _all_ steps below.
### Jira
https://issues.apache.org/jira/browse/AIRFLOW-3706
### Description
The "airflowDefaultTheme.css" file is created by webpack out of
bootstrap-theme.css, which is a "base" file. But this was loaded _after_
Airflow's main.css, meaning the `max-width` we set on `.tooltip-inner`
was being replaced by the default from bootstrap.
/cc @verdan
### Tests
**Before**:
**After**:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-3706) Tooltip Display Issue for a Task Instance
[ https://issues.apache.org/jira/browse/AIRFLOW-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-3706: --- Fix Version/s: (was: 1.10.2) 1.10.3 > Tooltip Display Issue for a Task Instance > - > > Key: AIRFLOW-3706 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3706 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.2 >Reporter: Felix Uellendall >Assignee: Ash Berlin-Taylor >Priority: Blocker > Fix For: 1.10.3 > > Attachments: 1.10.1-firefox.png, 1.10.1-safari.png, > 1.10.2b2-firefox.png, 1.10.2b2-safari.png > > > I am facing a tooltip display issue in 1.10.2b2 when hovering over a task > instance the tooltip rectangle that is being displayed is not automatically > resizing - for 1.10.1 it does. > I am using the RBAC UI. > I tested it with Firefox 64.0.2 (64-bit) and Safari Version 11.1.2 > (13605.3.8) on mac OS High Sierra Version 10.13.6 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4118) Instrument dagrun duration
[
https://issues.apache.org/jira/browse/AIRFLOW-4118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796608#comment-16796608
]
ASF GitHub Bot commented on AIRFLOW-4118:
-
milton0825 commented on pull request #4946: [AIRFLOW-4118] instrument DagRun
duration
URL: https://github.com/apache/airflow/pull/4946
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-4118
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Instrument dagrun duration
> --
>
> Key: AIRFLOW-4118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4118
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] milton0825 opened a new pull request #4946: [AIRFLOW-4118] instrument DagRun duration
milton0825 opened a new pull request #4946: [AIRFLOW-4118] instrument DagRun
duration
URL: https://github.com/apache/airflow/pull/4946
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-4118
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Reopened] (AIRFLOW-3706) Tooltip Display Issue for a Task Instance
[ https://issues.apache.org/jira/browse/AIRFLOW-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor reopened AIRFLOW-3706: Assignee: Ash Berlin-Taylor This was never resolved in 1.10.2 > Tooltip Display Issue for a Task Instance > - > > Key: AIRFLOW-3706 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3706 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.2 >Reporter: Felix Uellendall >Assignee: Ash Berlin-Taylor >Priority: Blocker > Fix For: 1.10.2 > > Attachments: 1.10.1-firefox.png, 1.10.1-safari.png, > 1.10.2b2-firefox.png, 1.10.2b2-safari.png > > > I am facing a tooltip display issue in 1.10.2b2 when hovering over a task > instance the tooltip rectangle that is being displayed is not automatically > resizing - for 1.10.1 it does. > I am using the RBAC UI. > I tested it with Firefox 64.0.2 (64-bit) and Safari Version 11.1.2 > (13605.3.8) on mac OS High Sierra Version 10.13.6 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4127) Azure Container Instance Hook _get_instance_view bug
[
https://issues.apache.org/jira/browse/AIRFLOW-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Asgeir Berland resolved AIRFLOW-4127.
-
Resolution: Fixed
Fix Version/s: 2.0.0
PR fix in https://github.com/apache/airflow/pull/4945
> Azure Container Instance Hook _get_instance_view bug
>
>
> Key: AIRFLOW-4127
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4127
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 2.0.0
>Reporter: Asgeir Berland
>Assignee: Asgeir Berland
>Priority: Major
> Labels: easyfix
> Fix For: 2.0.0
>
> Original Estimate: 20m
> Remaining Estimate: 20m
>
> Code:
> [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
>
> The ".current_state" needs to be stripped from the return call in this
> method. You can easily see why if you look at line107 and line129, which
> expects an "instance_view" and not "current_state".
> {code}
> return response.containers[0].instance_view.current_state
> ->
> return response.containers[0].instance_view
> {code}
> The current state of this bug makes it impossible to poll containers for
> status which is essential to detect when they have terminated their jobs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work stopped] (AIRFLOW-4127) Azure Container Instance Hook _get_instance_view bug
[
https://issues.apache.org/jira/browse/AIRFLOW-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on AIRFLOW-4127 stopped by Asgeir Berland.
---
> Azure Container Instance Hook _get_instance_view bug
>
>
> Key: AIRFLOW-4127
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4127
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 2.0.0
>Reporter: Asgeir Berland
>Assignee: Asgeir Berland
>Priority: Major
> Labels: easyfix
> Original Estimate: 20m
> Remaining Estimate: 20m
>
> Code:
> [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
>
> The ".current_state" needs to be stripped from the return call in this
> method. You can easily see why if you look at line107 and line129, which
> expects an "instance_view" and not "current_state".
> {code}
> return response.containers[0].instance_view.current_state
> ->
> return response.containers[0].instance_view
> {code}
> The current state of this bug makes it impossible to poll containers for
> status which is essential to detect when they have terminated their jobs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (AIRFLOW-4127) Azure Container Instance Hook _get_instance_view bug
[
https://issues.apache.org/jira/browse/AIRFLOW-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on AIRFLOW-4127 started by Asgeir Berland.
---
> Azure Container Instance Hook _get_instance_view bug
>
>
> Key: AIRFLOW-4127
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4127
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 2.0.0
>Reporter: Asgeir Berland
>Assignee: Asgeir Berland
>Priority: Major
> Labels: easyfix
> Original Estimate: 20m
> Remaining Estimate: 20m
>
> Code:
> [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
>
> The ".current_state" needs to be stripped from the return call in this
> method. You can easily see why if you look at line107 and line129, which
> expects an "instance_view" and not "current_state".
> {code}
> return response.containers[0].instance_view.current_state
> ->
> return response.containers[0].instance_view
> {code}
> The current state of this bug makes it impossible to poll containers for
> status which is essential to detect when they have terminated their jobs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4127) Azure Container Instance Hook _get_instance_view bug
[
https://issues.apache.org/jira/browse/AIRFLOW-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796603#comment-16796603
]
ASF GitHub Bot commented on AIRFLOW-4127:
-
Trollgeir commented on pull request #4945: [AIRFLOW-4127] _get_instance_view
returns instance view
URL: https://github.com/apache/airflow/pull/4945
### Jira
https://issues.apache.org/jira/browse/AIRFLOW-4127
### Description
Simple bugfix for method `_get_instance_view` that returned `current_view`
instead of an `instance_view`
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Azure Container Instance Hook _get_instance_view bug
>
>
> Key: AIRFLOW-4127
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4127
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 2.0.0
>Reporter: Asgeir Berland
>Assignee: Asgeir Berland
>Priority: Major
> Labels: easyfix
> Original Estimate: 20m
> Remaining Estimate: 20m
>
> Code:
> [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
>
> The ".current_state" needs to be stripped from the return call in this
> method. You can easily see why if you look at line107 and line129, which
> expects an "instance_view" and not "current_state".
> {code}
> return response.containers[0].instance_view.current_state
> ->
> return response.containers[0].instance_view
> {code}
> The current state of this bug makes it impossible to poll containers for
> status which is essential to detect when they have terminated their jobs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] Trollgeir opened a new pull request #4945: [AIRFLOW-4127] _get_instance_view returns instance view
Trollgeir opened a new pull request #4945: [AIRFLOW-4127] _get_instance_view
returns instance view
URL: https://github.com/apache/airflow/pull/4945
### Jira
https://issues.apache.org/jira/browse/AIRFLOW-4127
### Description
Simple bugfix for method `_get_instance_view` that returned `current_view`
instead of an `instance_view`
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Assigned] (AIRFLOW-4127) Azure Container Instance Hook _get_instance_view bug
[
https://issues.apache.org/jira/browse/AIRFLOW-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Asgeir Berland reassigned AIRFLOW-4127:
---
Assignee: Asgeir Berland
> Azure Container Instance Hook _get_instance_view bug
>
>
> Key: AIRFLOW-4127
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4127
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 2.0.0
>Reporter: Asgeir Berland
>Assignee: Asgeir Berland
>Priority: Major
> Labels: easyfix
> Original Estimate: 20m
> Remaining Estimate: 20m
>
> Code:
> [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
>
> The ".current_state" needs to be stripped from the return call in this
> method. You can easily see why if you look at line107 and line129, which
> expects an "instance_view" and not "current_state".
> {code}
> return response.containers[0].instance_view.current_state
> ->
> return response.containers[0].instance_view
> {code}
> The current state of this bug makes it impossible to poll containers for
> status which is essential to detect when they have terminated their jobs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4110) XSS issue with Search in Webserver
[ https://issues.apache.org/jira/browse/AIRFLOW-4110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-4110. Resolution: Duplicate > XSS issue with Search in Webserver > -- > > Key: AIRFLOW-4110 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4110 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Affects Versions: 1.10.2 >Reporter: Harri >Priority: Major > Labels: Search, Security > > There is an XSS issue with Search in Webserver. One can inject javascript to > search functionality. > > E.g. one can get javascript alert window injected: > GET > /airflow/admin/?search=hupqd%22%3e%3cscript%3ealert(1)%3c%2fscript%3eel10r > HTTP/1.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4129) Escape HTML in tooltips
[
https://issues.apache.org/jira/browse/AIRFLOW-4129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796596#comment-16796596
]
ASF GitHub Bot commented on AIRFLOW-4129:
-
ashb commented on pull request #4944: [AIRFLOW-4129] Escape HTML in generated
tooltips
URL: https://github.com/apache/airflow/pull/4944
Many of these properties aren't user generated, but it is better to be
safe
Make sure you have checked _all_ steps below.
### Jira
https://issues.apache.org/jira/browse/AIRFLOW-4129
### Description
Escape anything that we expect to be plain text when building tooltips.
### Tests
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Escape HTML in tooltips
> ---
>
> Key: AIRFLOW-4129
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4129
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.2
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] ashb opened a new pull request #4944: [AIRFLOW-4129] Escape HTML in generated tooltips
ashb opened a new pull request #4944: [AIRFLOW-4129] Escape HTML in generated
tooltips
URL: https://github.com/apache/airflow/pull/4944
Many of these properties aren't user generated, but it is better to be
safe
Make sure you have checked _all_ steps below.
### Jira
https://issues.apache.org/jira/browse/AIRFLOW-4129
### Description
Escape anything that we expect to be plain text when building tooltips.
### Tests
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-4129) Escape HTML in tooltips
Ash Berlin-Taylor created AIRFLOW-4129: -- Summary: Escape HTML in tooltips Key: AIRFLOW-4129 URL: https://issues.apache.org/jira/browse/AIRFLOW-4129 Project: Apache Airflow Issue Type: Bug Affects Versions: 1.10.2 Reporter: Ash Berlin-Taylor Assignee: Ash Berlin-Taylor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4127) Azure Container Instance Hook _get_instance_view bug
Asgeir Berland created AIRFLOW-4127: --- Summary: Azure Container Instance Hook _get_instance_view bug Key: AIRFLOW-4127 URL: https://issues.apache.org/jira/browse/AIRFLOW-4127 Project: Apache Airflow Issue Type: Bug Components: hooks Affects Versions: 2.0.0 Reporter: Asgeir Berland Code: [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116] The ".current_state" needs to be stripped from the return call in this method. You can easily see why if you look at line107 and line129, which expects an "instance_view" and not "current_state". The current state of this bug makes it impossible to poll containers for status which is essential to detect when they have terminated their jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4127) Azure Container Instance Hook _get_instance_view bug
[
https://issues.apache.org/jira/browse/AIRFLOW-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Asgeir Berland updated AIRFLOW-4127:
Description:
Code:
[https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
The ".current_state" needs to be stripped from the return call in this method.
You can easily see why if you look at line107 and line129, which expects an
"instance_view" and not "current_state".
{{{code: python}}}
return response.containers[0].instance_view.current_state
->
return response.containers[0].instance_view
{code}
The current state of this bug makes it impossible to poll containers for status
which is essential to detect when they have terminated their jobs.
was:
Code:
[https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
{{{noformat}}}
return response.containers[0].instance_view.current_state
{{{noformat}}}
->
{{{noformat}}}
return response.containers[0].instance_view
{{{noformat}}}
The ".current_state" needs to be stripped from the return call in this method.
You can easily see why if you look at line107 and line129, which expects an
"instance_view" and not "current_state".
The current state of this bug makes it impossible to poll containers for status
which is essential to detect when they have terminated their jobs.
> Azure Container Instance Hook _get_instance_view bug
>
>
> Key: AIRFLOW-4127
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4127
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 2.0.0
>Reporter: Asgeir Berland
>Priority: Major
> Labels: easyfix
> Original Estimate: 20m
> Remaining Estimate: 20m
>
> Code:
> [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
>
> The ".current_state" needs to be stripped from the return call in this
> method. You can easily see why if you look at line107 and line129, which
> expects an "instance_view" and not "current_state".
> {{{code: python}}}
> return response.containers[0].instance_view.current_state
> ->
> return response.containers[0].instance_view
> {code}
> The current state of this bug makes it impossible to poll containers for
> status which is essential to detect when they have terminated their jobs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] feng-tao edited a comment on issue #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 …
feng-tao edited a comment on issue #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 … URL: https://github.com/apache/airflow/pull/4941#issuecomment-474612856 @dimberman , I think as a committer, we could restart / trigger individual test(e.g only the failed test) instead of doing a full rebuild. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 …
feng-tao commented on issue #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 … URL: https://github.com/apache/airflow/pull/4941#issuecomment-474612856 @dimberman , I think as a committer, we could restart individual test(e.g only the failed test) instead of doing a full rebuild. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman commented on issue #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 …
dimberman commented on issue #4941: [AIRFLOW-4123] Add Exception handling for _change_state method in K8 … URL: https://github.com/apache/airflow/pull/4941#issuecomment-474612454 @feng-tao I now have the squash + merge ability. I'm gonna re-run the tests and if/when they pass I'll merge (first one, woo!). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-4128) Timezone transferring in macro doesn't work
Takako Ohshima created AIRFLOW-4128:
---
Summary: Timezone transferring in macro doesn't work
Key: AIRFLOW-4128
URL: https://issues.apache.org/jira/browse/AIRFLOW-4128
Project: Apache Airflow
Issue Type: Bug
Components: DAG
Affects Versions: 1.10.1, 1.10.0
Environment: Cloud Composer
Reporter: Takako Ohshima
On Airflow 1.9, it worked like this timezone transferring like below.
But 1.10.x seems doesn’t work this way with `Error rendering template:
'NoneType' object has no attribute 'total_seconds'` error.
{code:java}
bash_command="echo {{
execution_date.replace(tzinfo=macros.dateutil.tz.gettz('US/Pacific')) }}"
bigquery_to_datastore_task = BashOperator(
dag=dag,
task_id="bigquery_to_datastore",
bash_command=bash_command,
)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4127) Azure Container Instance Hook _get_instance_view bug
[
https://issues.apache.org/jira/browse/AIRFLOW-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Asgeir Berland updated AIRFLOW-4127:
Description:
Code:
[https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
The ".current_state" needs to be stripped from the return call in this method.
You can easily see why if you look at line107 and line129, which expects an
"instance_view" and not "current_state".
{code}
return response.containers[0].instance_view.current_state
->
return response.containers[0].instance_view
{code}
The current state of this bug makes it impossible to poll containers for status
which is essential to detect when they have terminated their jobs.
was:
Code:
[https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
The ".current_state" needs to be stripped from the return call in this method.
You can easily see why if you look at line107 and line129, which expects an
"instance_view" and not "current_state".
{{{code: python}}}
return response.containers[0].instance_view.current_state
->
return response.containers[0].instance_view
{code}
The current state of this bug makes it impossible to poll containers for status
which is essential to detect when they have terminated their jobs.
> Azure Container Instance Hook _get_instance_view bug
>
>
> Key: AIRFLOW-4127
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4127
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 2.0.0
>Reporter: Asgeir Berland
>Priority: Major
> Labels: easyfix
> Original Estimate: 20m
> Remaining Estimate: 20m
>
> Code:
> [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
>
> The ".current_state" needs to be stripped from the return call in this
> method. You can easily see why if you look at line107 and line129, which
> expects an "instance_view" and not "current_state".
> {code}
> return response.containers[0].instance_view.current_state
> ->
> return response.containers[0].instance_view
> {code}
> The current state of this bug makes it impossible to poll containers for
> status which is essential to detect when they have terminated their jobs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4127) Azure Container Instance Hook _get_instance_view bug
[
https://issues.apache.org/jira/browse/AIRFLOW-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Asgeir Berland updated AIRFLOW-4127:
Description:
Code:
[https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
{{{noformat}}}
return response.containers[0].instance_view.current_state
{{{noformat}}}
->
{{{noformat}}}
return response.containers[0].instance_view
{{{noformat}}}
The ".current_state" needs to be stripped from the return call in this method.
You can easily see why if you look at line107 and line129, which expects an
"instance_view" and not "current_state".
The current state of this bug makes it impossible to poll containers for status
which is essential to detect when they have terminated their jobs.
was:
Code:
[https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
{{{noformat}}}
return response.containers[0].instance_view.current_state
->
return response.containers[0].instance_view
{{{noformat}}}
The ".current_state" needs to be stripped from the return call in this method.
You can easily see why if you look at line107 and line129, which expects an
"instance_view" and not "current_state".
The current state of this bug makes it impossible to poll containers for status
which is essential to detect when they have terminated their jobs.
> Azure Container Instance Hook _get_instance_view bug
>
>
> Key: AIRFLOW-4127
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4127
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 2.0.0
>Reporter: Asgeir Berland
>Priority: Major
> Labels: easyfix
> Original Estimate: 20m
> Remaining Estimate: 20m
>
> Code:
> [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
>
> {{{noformat}}}
> return response.containers[0].instance_view.current_state
> {{{noformat}}}
> ->
> {{{noformat}}}
> return response.containers[0].instance_view
> {{{noformat}}}
>
> The ".current_state" needs to be stripped from the return call in this
> method. You can easily see why if you look at line107 and line129, which
> expects an "instance_view" and not "current_state".
> The current state of this bug makes it impossible to poll containers for
> status which is essential to detect when they have terminated their jobs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4127) Azure Container Instance Hook _get_instance_view bug
[
https://issues.apache.org/jira/browse/AIRFLOW-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Asgeir Berland updated AIRFLOW-4127:
Description:
Code:
[https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
{{{noformat}}}
return response.containers[0].instance_view.current_state
->
return response.containers[0].instance_view
{{{noformat}}}
The ".current_state" needs to be stripped from the return call in this method.
You can easily see why if you look at line107 and line129, which expects an
"instance_view" and not "current_state".
The current state of this bug makes it impossible to poll containers for status
which is essential to detect when they have terminated their jobs.
was:
Code:
[https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
```return response.containers[0].instance_view.current_state```
->
```return response.containers[0].instance_view```
The ".current_state" needs to be stripped from the return call in this method.
You can easily see why if you look at line107 and line129, which expects an
"instance_view" and not "current_state".
The current state of this bug makes it impossible to poll containers for status
which is essential to detect when they have terminated their jobs.
> Azure Container Instance Hook _get_instance_view bug
>
>
> Key: AIRFLOW-4127
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4127
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 2.0.0
>Reporter: Asgeir Berland
>Priority: Major
> Labels: easyfix
> Original Estimate: 20m
> Remaining Estimate: 20m
>
> Code:
> [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
>
> {{{noformat}}}
> return response.containers[0].instance_view.current_state
> ->
> return response.containers[0].instance_view
> {{{noformat}}}
>
> The ".current_state" needs to be stripped from the return call in this
> method. You can easily see why if you look at line107 and line129, which
> expects an "instance_view" and not "current_state".
> The current state of this bug makes it impossible to poll containers for
> status which is essential to detect when they have terminated their jobs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4127) Azure Container Instance Hook _get_instance_view bug
[ https://issues.apache.org/jira/browse/AIRFLOW-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asgeir Berland updated AIRFLOW-4127: Description: Code: [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116] ```return response.containers[0].instance_view.current_state``` -> ```return response.containers[0].instance_view``` The ".current_state" needs to be stripped from the return call in this method. You can easily see why if you look at line107 and line129, which expects an "instance_view" and not "current_state". The current state of this bug makes it impossible to poll containers for status which is essential to detect when they have terminated their jobs. was: Code: [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116] The ".current_state" needs to be stripped from the return call in this method. You can easily see why if you look at line107 and line129, which expects an "instance_view" and not "current_state". The current state of this bug makes it impossible to poll containers for status which is essential to detect when they have terminated their jobs. > Azure Container Instance Hook _get_instance_view bug > > > Key: AIRFLOW-4127 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4127 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 2.0.0 >Reporter: Asgeir Berland >Priority: Major > Labels: easyfix > Original Estimate: 20m > Remaining Estimate: 20m > > Code: > [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116] > > ```return response.containers[0].instance_view.current_state``` > -> > ```return response.containers[0].instance_view``` > The ".current_state" needs to be stripped from the return call in this > method. You can easily see why if you look at line107 and line129, which > expects an "instance_view" and not "current_state". > The current state of this bug makes it impossible to poll containers for > status which is essential to detect when they have terminated their jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4127) Azure Container Instance Hook _get_instance_view bug
[ https://issues.apache.org/jira/browse/AIRFLOW-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asgeir Berland updated AIRFLOW-4127: Remaining Estimate: 20m (was: 10m) Original Estimate: 20m (was: 10m) > Azure Container Instance Hook _get_instance_view bug > > > Key: AIRFLOW-4127 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4127 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 2.0.0 >Reporter: Asgeir Berland >Priority: Major > Labels: easyfix > Original Estimate: 20m > Remaining Estimate: 20m > > Code: > [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116] > The ".current_state" needs to be stripped from the return call in this > method. You can easily see why if you look at line107 and line129, which > expects an "instance_view" and not "current_state". > The current state of this bug makes it impossible to poll containers for > status which is essential to detect when they have terminated their jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] morgendave commented on issue #4101: [AIRFLOW-3272] Add base grpc hook
morgendave commented on issue #4101: [AIRFLOW-3272] Add base grpc hook
URL: https://github.com/apache/airflow/pull/4101#issuecomment-474608670
> Not sure if this is due to my misconfiguration, but before this commit my
setup works while after it, I get a jinja2 error of `extra__grpc__auth_type`
not defined when accessing connections from the new www fab UI.
>
> ```
> / ( () ) \___
> /( ( ( ) _)) ) )\
>(( ( )() ) ( ) )
> ((/ ( _( ) ( _) ) ( () ) )
> ( ( ( (_) ((( ) .((_ ) . )_
>( ( )( ( )) ) . ) ( )
> ( ( ( ( ) ( _ ( _) ). ) . ) ) ( )
> ( ( ( ) ( ) ( )) ) _)( ) ) )
> ( ( ( \ ) ((_ ( ) ( ) ) ) ) )) ( )
> ( ( ( ( (_ ( ) ( _) ) ( ) ) )
> ( ( ( ( ( ) (_ ) ) ) _) ) _( ( )
> (( ( )(( _) _) _(_ ( (_ )
>(_((__(_(__(( ( ( | ) ) ) )_))__))_)___)
>((__)\\||lll|l||/// \_))
> ( /(/ ( ) ) )\ )
> (( ( ( | | ) ) )\ )
>( /(| / ( )) ) ) )) )
> ( ( _(|)_) )
> ( ||\(|(|)|/|| )
> (|(||(||))
> ( //|/l|||)|\\ \ )
> (/ / // /|//\\ \ \ \ _)
>
---
> Node: airflow-staging-web-7fd889645d-6bnt2
>
---
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2292,
in wsgi_app
> response = self.full_dispatch_request()
> File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1815,
in full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1718,
in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 35,
in reraise
> raise value
> File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1813,
in full_dispatch_request
> rv = self.dispatch_request()
> File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1799,
in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File
"/usr/local/lib/python3.7/site-packages/flask_appbuilder/security/decorators.py",
line 26, in wraps
> return f(self, *args, **kwargs)
> File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/views.py",
line 512, in add
> widgets=widget)
> File
"/usr/local/lib/python3.7/site-packages/flask_appbuilder/baseviews.py", line
161, in render_template
> return render_template(template, **dict(list(kwargs.items()) +
list(self.extra_args.items(
> File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line
135, in render_template
> context, ctx.app)
> File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line
117, in _render
> rv = template.render(context)
> File "/usr/local/lib/python3.7/site-packages/jinja2/asyncsupport.py",
line 76, in render
> return original_render(self, *args, **kwargs)
> File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py",
line 1008, in render
> return self.environment.handle_exception(exc_info, True)
> File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py",
line 780, in handle_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/local/lib/python3.7/site-packages/jinja2/_compat.py", line
37, in reraise
> raise value.with_traceback(tb)
> File
"/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/conn_create.html",
line 18, in top-level template code
> {% extends 'appbuilder/general/model/add.html' %}
> File
"/usr/local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/general/model/add.html",
line 2, in top-level template code
> {% import 'appbuilder/general/lib.html' as lib %}
> File
"/usr/local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/base.html",
line 1, in top-level template code
> {% extends base_template %}
> File
"/usr/local/lib/python3.7/site-packages/airflow/www/templates/appbuilder/baselayout.html",
line 19, in top-level template code
> {% import 'appbuilder/baselib.html'
[GitHub] [airflow] feng-tao commented on a change in pull request #4889: [AIRFLOW-4057] statsd should escape invalid characters
feng-tao commented on a change in pull request #4889: [AIRFLOW-4057] statsd
should escape invalid characters
URL: https://github.com/apache/airflow/pull/4889#discussion_r267120874
##
File path: airflow/stats.py
##
@@ -0,0 +1,124 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from functools import wraps
+import logging
+import socket
+import string
+import textwrap
+from typing import Any
+
+from airflow import configuration as conf
+from airflow.exceptions import InvalidStatsNameException
+
+log = logging.getLogger(__name__)
+
+
+class DummyStatsLogger(object):
+@classmethod
+def incr(cls, stat, count=1, rate=1):
+pass
+
+@classmethod
+def decr(cls, stat, count=1, rate=1):
+pass
+
+@classmethod
+def gauge(cls, stat, value, rate=1, delta=False):
+pass
+
+@classmethod
+def timing(cls, stat, dt):
+pass
+
+
+# Only characters in the character set are considered valid
+# for the stat_name if stat_name_default_handler is used.
+ALLOWED_CHARACTERS = set(string.ascii_letters + string.digits + '_.-')
+
+
+def stat_name_default_handler(stat_name, max_length=250):
+if not isinstance(stat_name, str):
+raise InvalidStatsNameException('The stat_name has to be a string')
+if len(stat_name) > max_length:
+raise InvalidStatsNameException(textwrap.dedent("""\
+The stat_name ({stat_name}) has to be less than {max_length}
characters.
+""".format(stat_name=stat_name, max_length=max_length)))
+if not all((c in ALLOWED_CHARACTERS) for c in stat_name):
+raise InvalidStatsNameException(textwrap.dedent("""\
+The stat name ({stat_name}) has to be composed with characters in
+{allowed_characters}.
+""".format(stat_name=stat_name,
+ allowed_characters=ALLOWED_CHARACTERS)))
+return stat_name
+
+
+def validate_stat(f):
+@wraps(f)
+def wrapper(stat, *args, **kwargs):
+try:
+from airflow.plugins_manager import stat_name_handler
+if stat_name_handler:
+handle_stat_name_func = stat_name_handler
+else:
+handle_stat_name_func = stat_name_default_handler
+stat_name = handle_stat_name_func(stat)
+except Exception as err:
+log.warning('Invalid stat name: {stat}.'.format(stat=stat), err)
+return
Review comment:
Ok, so I think the pr title should still be "handle invalid char" in this
case, WDYT @ashb ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feng-tao commented on a change in pull request #4889: [AIRFLOW-4057] statsd should escape invalid characters
feng-tao commented on a change in pull request #4889: [AIRFLOW-4057] statsd
should escape invalid characters
URL: https://github.com/apache/airflow/pull/4889#discussion_r267120874
##
File path: airflow/stats.py
##
@@ -0,0 +1,124 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from functools import wraps
+import logging
+import socket
+import string
+import textwrap
+from typing import Any
+
+from airflow import configuration as conf
+from airflow.exceptions import InvalidStatsNameException
+
+log = logging.getLogger(__name__)
+
+
+class DummyStatsLogger(object):
+@classmethod
+def incr(cls, stat, count=1, rate=1):
+pass
+
+@classmethod
+def decr(cls, stat, count=1, rate=1):
+pass
+
+@classmethod
+def gauge(cls, stat, value, rate=1, delta=False):
+pass
+
+@classmethod
+def timing(cls, stat, dt):
+pass
+
+
+# Only characters in the character set are considered valid
+# for the stat_name if stat_name_default_handler is used.
+ALLOWED_CHARACTERS = set(string.ascii_letters + string.digits + '_.-')
+
+
+def stat_name_default_handler(stat_name, max_length=250):
+if not isinstance(stat_name, str):
+raise InvalidStatsNameException('The stat_name has to be a string')
+if len(stat_name) > max_length:
+raise InvalidStatsNameException(textwrap.dedent("""\
+The stat_name ({stat_name}) has to be less than {max_length}
characters.
+""".format(stat_name=stat_name, max_length=max_length)))
+if not all((c in ALLOWED_CHARACTERS) for c in stat_name):
+raise InvalidStatsNameException(textwrap.dedent("""\
+The stat name ({stat_name}) has to be composed with characters in
+{allowed_characters}.
+""".format(stat_name=stat_name,
+ allowed_characters=ALLOWED_CHARACTERS)))
+return stat_name
+
+
+def validate_stat(f):
+@wraps(f)
+def wrapper(stat, *args, **kwargs):
+try:
+from airflow.plugins_manager import stat_name_handler
+if stat_name_handler:
+handle_stat_name_func = stat_name_handler
+else:
+handle_stat_name_func = stat_name_default_handler
+stat_name = handle_stat_name_func(stat)
+except Exception as err:
+log.warning('Invalid stat name: {stat}.'.format(stat=stat), err)
+return
Review comment:
Ok, so I think the pr title should be still "handle invalid char" in this
case, WDYT @ashb ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] milton0825 commented on a change in pull request #4889: [AIRFLOW-4057] statsd should escape invalid characters
milton0825 commented on a change in pull request #4889: [AIRFLOW-4057] statsd
should escape invalid characters
URL: https://github.com/apache/airflow/pull/4889#discussion_r267119663
##
File path: airflow/stats.py
##
@@ -0,0 +1,124 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from functools import wraps
+import logging
+import socket
+import string
+import textwrap
+from typing import Any
+
+from airflow import configuration as conf
+from airflow.exceptions import InvalidStatsNameException
+
+log = logging.getLogger(__name__)
+
+
+class DummyStatsLogger(object):
+@classmethod
+def incr(cls, stat, count=1, rate=1):
+pass
+
+@classmethod
+def decr(cls, stat, count=1, rate=1):
+pass
+
+@classmethod
+def gauge(cls, stat, value, rate=1, delta=False):
+pass
+
+@classmethod
+def timing(cls, stat, dt):
+pass
+
+
+# Only characters in the character set are considered valid
+# for the stat_name if stat_name_default_handler is used.
+ALLOWED_CHARACTERS = set(string.ascii_letters + string.digits + '_.-')
+
+
+def stat_name_default_handler(stat_name, max_length=250):
+if not isinstance(stat_name, str):
+raise InvalidStatsNameException('The stat_name has to be a string')
+if len(stat_name) > max_length:
+raise InvalidStatsNameException(textwrap.dedent("""\
+The stat_name ({stat_name}) has to be less than {max_length}
characters.
+""".format(stat_name=stat_name, max_length=max_length)))
+if not all((c in ALLOWED_CHARACTERS) for c in stat_name):
+raise InvalidStatsNameException(textwrap.dedent("""\
+The stat name ({stat_name}) has to be composed with characters in
+{allowed_characters}.
+""".format(stat_name=stat_name,
+ allowed_characters=ALLOWED_CHARACTERS)))
+return stat_name
+
+
+def validate_stat(f):
+@wraps(f)
+def wrapper(stat, *args, **kwargs):
+try:
+from airflow.plugins_manager import stat_name_handler
+if stat_name_handler:
+handle_stat_name_func = stat_name_handler
+else:
+handle_stat_name_func = stat_name_default_handler
+stat_name = handle_stat_name_func(stat)
+except Exception as err:
+log.warning('Invalid stat name: {stat}.'.format(stat=stat), err)
+return
Review comment:
Well @ashb changed the title to "escape invalid char". It was "handle
invalid char" when I make the PR initially.
The `stat_name_default_handler` will log warning about that the `stat_name`
is invalid and not sending stats at all. If the user want to escape invalid
characters in stat_name, they can override it with a custom method to define
how they want to escape it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feng-tao commented on a change in pull request #4889: [AIRFLOW-4057] statsd should escape invalid characters
feng-tao commented on a change in pull request #4889: [AIRFLOW-4057] statsd
should escape invalid characters
URL: https://github.com/apache/airflow/pull/4889#discussion_r267118112
##
File path: airflow/stats.py
##
@@ -0,0 +1,124 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from functools import wraps
+import logging
+import socket
+import string
+import textwrap
+from typing import Any
+
+from airflow import configuration as conf
+from airflow.exceptions import InvalidStatsNameException
+
+log = logging.getLogger(__name__)
+
+
+class DummyStatsLogger(object):
+@classmethod
+def incr(cls, stat, count=1, rate=1):
+pass
+
+@classmethod
+def decr(cls, stat, count=1, rate=1):
+pass
+
+@classmethod
+def gauge(cls, stat, value, rate=1, delta=False):
+pass
+
+@classmethod
+def timing(cls, stat, dt):
+pass
+
+
+# Only characters in the character set are considered valid
+# for the stat_name if stat_name_default_handler is used.
+ALLOWED_CHARACTERS = set(string.ascii_letters + string.digits + '_.-')
+
+
+def stat_name_default_handler(stat_name, max_length=250):
+if not isinstance(stat_name, str):
+raise InvalidStatsNameException('The stat_name has to be a string')
+if len(stat_name) > max_length:
+raise InvalidStatsNameException(textwrap.dedent("""\
+The stat_name ({stat_name}) has to be less than {max_length}
characters.
+""".format(stat_name=stat_name, max_length=max_length)))
+if not all((c in ALLOWED_CHARACTERS) for c in stat_name):
+raise InvalidStatsNameException(textwrap.dedent("""\
+The stat name ({stat_name}) has to be composed with characters in
+{allowed_characters}.
+""".format(stat_name=stat_name,
+ allowed_characters=ALLOWED_CHARACTERS)))
+return stat_name
+
+
+def validate_stat(f):
+@wraps(f)
+def wrapper(stat, *args, **kwargs):
+try:
+from airflow.plugins_manager import stat_name_handler
+if stat_name_handler:
+handle_stat_name_func = stat_name_handler
+else:
+handle_stat_name_func = stat_name_default_handler
+stat_name = handle_stat_name_func(stat)
+except Exception as err:
+log.warning('Invalid stat name: {stat}.'.format(stat=stat), err)
+return
Review comment:
I see
@ashb's point now, I guess the confusion is that the pr title is "escape
invalid char" while it is throw exception. Then you swallow the exception here.
I wonder if the stat name is invalid what will happen @milton0825 ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
