[GitHub] [airflow] jblakeman edited a comment on issue #8484: Airflow 1.10.7+ suppresses Operator logs

2020-05-18 Thread GitBox
jblakeman edited a comment on issue #8484: URL: https://github.com/apache/airflow/issues/8484#issuecomment-630315582 Had a chance to skim through #6627 and I could be mistaken (have not found the time to grok through all of the moving parts yet) but it seems like using `os._exit` in the

[GitHub] [airflow] yuqian90 commented on a change in pull request #8867: Add argument to BaseSensorOperator to make respect the trigger_rule in downstream tasks.

2020-05-18 Thread GitBox
yuqian90 commented on a change in pull request #8867: URL: https://github.com/apache/airflow/pull/8867#discussion_r427012430 ## File path: airflow/sensors/base_sensor_operator.py ## @@ -131,6 +134,19 @@ def execute(self, context: Dict) -> Any: try_number += 1

[GitHub] [airflow] mik-laj commented on a change in pull request #8808: Support YAML input for CloudBuildCreateOperator

2020-05-18 Thread GitBox
mik-laj commented on a change in pull request #8808: URL: https://github.com/apache/airflow/pull/8808#discussion_r427014320 ## File path: airflow/providers/google/cloud/example_dags/example_cloud_build.py ## @@ -99,6 +102,12 @@ task_id="create_build_from_repo_result",

[GitHub] [airflow] mik-laj commented on a change in pull request #8808: Support YAML input for CloudBuildCreateOperator

2020-05-18 Thread GitBox
mik-laj commented on a change in pull request #8808: URL: https://github.com/apache/airflow/pull/8808#discussion_r427014320 ## File path: airflow/providers/google/cloud/example_dags/example_cloud_build.py ## @@ -99,6 +102,12 @@ task_id="create_build_from_repo_result",

[GitHub] [airflow] mik-laj commented on a change in pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library

2020-05-18 Thread GitBox
mik-laj commented on a change in pull request #8575: URL: https://github.com/apache/airflow/pull/8575#discussion_r427011527 ## File path: airflow/providers/google/cloud/example_dags/example_cloud_build.py ## @@ -60,7 +87,7 @@ # [START

[jira] [Commented] (AIRFLOW-6981) Move Google Cloud Build from Discovery API to Python Library

2020-05-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110823#comment-17110823 ] ASF GitHub Bot commented on AIRFLOW-6981: - mik-laj commented on a change in pull request #8575:

[jira] [Commented] (AIRFLOW-6981) Move Google Cloud Build from Discovery API to Python Library

2020-05-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110821#comment-17110821 ] ASF GitHub Bot commented on AIRFLOW-6981: - mik-laj commented on pull request #8575: URL:

[GitHub] [airflow] mik-laj commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library

2020-05-18 Thread GitBox
mik-laj commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-630558831 I tried to run system tests and unfortunately none of the tasks was successful. ``` [2020-05-19 03:29:06,688] {taskinstance.py:719} INFO - Dependencies all

[jira] [Commented] (AIRFLOW-6981) Move Google Cloud Build from Discovery API to Python Library

2020-05-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110816#comment-17110816 ] ASF GitHub Bot commented on AIRFLOW-6981: - mik-laj commented on a change in pull request #8575:

[GitHub] [airflow] mik-laj commented on a change in pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library

2020-05-18 Thread GitBox
mik-laj commented on a change in pull request #8575: URL: https://github.com/apache/airflow/pull/8575#discussion_r427008238 ## File path: airflow/providers/google/cloud/operators/cloud_build.py ## @@ -15,186 +15,678 @@ # KIND, either express or implied. See the License for

[jira] [Commented] (AIRFLOW-6569) Broken sentry integration

2020-05-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110815#comment-17110815 ] ASF GitHub Bot commented on AIRFLOW-6569: - maxcountryman commented on pull request #7232: URL:

[GitHub] [airflow] maxcountryman commented on pull request #7232: [AIRFLOW-6569] Flush pending Sentry exceptions before exiting forked process

2020-05-18 Thread GitBox
maxcountryman commented on pull request #7232: URL: https://github.com/apache/airflow/pull/7232#issuecomment-630555724 I believe the bug this patch addresses is actively preventing us from receiving alerts in production.

[airflow] branch master updated (2121f49 -> 841d816)

2020-05-18 Thread kamilbregula
This is an automated email from the ASF dual-hosted git repository. kamilbregula pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 2121f49 Avoid failure on transient requirements in CI image (#8892) add 841d816 Allow setting the

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8824: Allow setting the pooling time in DLPHook

2020-05-18 Thread GitBox
boring-cyborg[bot] commented on pull request #8824: URL: https://github.com/apache/airflow/pull/8824#issuecomment-630548745 Awesome work, congrats on your first merged pull request! This is an automated message from the

[GitHub] [airflow] mik-laj merged pull request #8824: Allow setting the pooling time in DLPHook

2020-05-18 Thread GitBox
mik-laj merged pull request #8824: URL: https://github.com/apache/airflow/pull/8824 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[jira] [Commented] (AIRFLOW-6290) Create guide for GKE service operator

2020-05-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110801#comment-17110801 ] ASF GitHub Bot commented on AIRFLOW-6290: - mik-laj commented on a change in pull request #8883:

[GitHub] [airflow] mik-laj commented on a change in pull request #8883: [AIRFLOW-6290] Create guide for GKE operators

2020-05-18 Thread GitBox
mik-laj commented on a change in pull request #8883: URL: https://github.com/apache/airflow/pull/8883#discussion_r426999884 ## File path: docs/howto/operator/gcp/kubernetes_engine.rst ## @@ -0,0 +1,122 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or

[jira] [Commented] (AIRFLOW-6290) Create guide for GKE service operator

2020-05-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110799#comment-17110799 ] ASF GitHub Bot commented on AIRFLOW-6290: - mik-laj commented on a change in pull request #8883:

[GitHub] [airflow] mik-laj commented on a change in pull request #8883: [AIRFLOW-6290] Create guide for GKE operators

2020-05-18 Thread GitBox
mik-laj commented on a change in pull request #8883: URL: https://github.com/apache/airflow/pull/8883#discussion_r426999545 ## File path: docs/howto/operator/gcp/kubernetes_engine.rst ## @@ -0,0 +1,122 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or

[jira] [Commented] (AIRFLOW-6290) Create guide for GKE service operator

2020-05-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110797#comment-17110797 ] ASF GitHub Bot commented on AIRFLOW-6290: - mik-laj commented on a change in pull request #8883:

[GitHub] [airflow] mik-laj commented on a change in pull request #8883: [AIRFLOW-6290] Create guide for GKE operators

2020-05-18 Thread GitBox
mik-laj commented on a change in pull request #8883: URL: https://github.com/apache/airflow/pull/8883#discussion_r426998501 ## File path: docs/howto/operator/gcp/kubernetes_engine.rst ## @@ -0,0 +1,122 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or

[GitHub] [airflow] listenLearning opened a new issue #8906: make success failed

2020-05-18 Thread GitBox
listenLearning opened a new issue #8906: URL: https://github.com/apache/airflow/issues/8906 I have recently installed a standalone version of airflow, version 1.10, then I created a dag, the task time is 2020-01-01-01, but maybe some tasks I want to start from the same day, instead of

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8906: make success failed

2020-05-18 Thread GitBox
boring-cyborg[bot] commented on issue #8906: URL: https://github.com/apache/airflow/issues/8906#issuecomment-630542345 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] mik-laj commented on a change in pull request #8883: [AIRFLOW-6290] Create guide for GKE operators

2020-05-18 Thread GitBox
mik-laj commented on a change in pull request #8883: URL: https://github.com/apache/airflow/pull/8883#discussion_r426994650 ## File path: docs/howto/operator/gcp/kubernetes_engine.rst ## @@ -0,0 +1,122 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or

[jira] [Commented] (AIRFLOW-6290) Create guide for GKE service operator

2020-05-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110786#comment-17110786 ] ASF GitHub Bot commented on AIRFLOW-6290: - mik-laj commented on a change in pull request #8883:

[GitHub] [airflow] mik-laj commented on a change in pull request #8883: [AIRFLOW-6290] Create guide for GKE operators

2020-05-18 Thread GitBox
mik-laj commented on a change in pull request #8883: URL: https://github.com/apache/airflow/pull/8883#discussion_r426994046 ## File path: docs/howto/operator/gcp/kubernetes_engine.rst ## @@ -0,0 +1,122 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or

[jira] [Commented] (AIRFLOW-6290) Create guide for GKE service operator

2020-05-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110782#comment-17110782 ] ASF GitHub Bot commented on AIRFLOW-6290: - mik-laj commented on a change in pull request #8883:

[GitHub] [airflow] enriqueayala commented on issue #8839: "Dag Runs" UI page shows broken State after enabled DAGs serialization

2020-05-18 Thread GitBox
enriqueayala commented on issue #8839: URL: https://github.com/apache/airflow/issues/8839#issuecomment-630520178 Having same issue for versions 1.10.10/ 1.10.9 and store_serialized_dags=False/True . No solution found yet..

[jira] [Commented] (AIRFLOW-6402) SSH operator exception - 'NoneType' object has no attribute 'startswith'

2020-05-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110754#comment-17110754 ] ASF GitHub Bot commented on AIRFLOW-6402: - stale[bot] commented on pull request #7008: URL:

[jira] [Commented] (AIRFLOW-6983) Disabled connection pool in CLI might be harmful

2020-05-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110752#comment-17110752 ] ASF GitHub Bot commented on AIRFLOW-6983: - stale[bot] commented on pull request #7622: URL:

[GitHub] [airflow] stale[bot] commented on pull request #7008: [AIRFLOW-6402] fix ssh_operator default value

2020-05-18 Thread GitBox
stale[bot] commented on pull request #7008: URL: https://github.com/apache/airflow/pull/7008#issuecomment-630516843 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[GitHub] [airflow] stale[bot] commented on pull request #7622: [AIRFLOW-6983] Configure connection pool disabling in CLI

2020-05-18 Thread GitBox
stale[bot] commented on pull request #7622: URL: https://github.com/apache/airflow/pull/7622#issuecomment-630516836 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jira] [Commented] (AIRFLOW-6404) Add ANSI color support

2020-05-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110753#comment-17110753 ] ASF GitHub Bot commented on AIRFLOW-6404: - stale[bot] commented on pull request #6998: URL:

[GitHub] [airflow] stale[bot] commented on pull request #6998: [AIRFLOW-6404] - ANSI color support for WebUI logger

2020-05-18 Thread GitBox
stale[bot] commented on pull request #6998: URL: https://github.com/apache/airflow/pull/6998#issuecomment-630516837 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8905: Logging with Custom Filenames

2020-05-18 Thread GitBox
boring-cyborg[bot] commented on issue #8905: URL: https://github.com/apache/airflow/issues/8905#issuecomment-630466226 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] athenawisdoms opened a new issue #8905: Logging with Custom Filenames

2020-05-18 Thread GitBox
athenawisdoms opened a new issue #8905: URL: https://github.com/apache/airflow/issues/8905 I am using Apache Airflow 1.10.10 to run several Python scripts in a DAG via the `BashOperator`. The logs are currently written to `/usr/airflow/logs`. Is it possible to configure Airflow

[GitHub] [airflow] ashb commented on pull request #8836: WIP - Add 3.8 to the test matrices.

2020-05-18 Thread GitBox
ashb commented on pull request #8836: URL: https://github.com/apache/airflow/pull/8836#issuecomment-630459846 Pymssql is already "replaced" by odbc in #6850, so I'd suggest just updating the deps etc so we don't try to install mssql on Py3.8, and skip those tests on 3.8 too.

[GitHub] [airflow] VijayantSoni commented on a change in pull request #8701: Adding ElastiCache Hook for creating, describing and deleting replication groups

2020-05-18 Thread GitBox
VijayantSoni commented on a change in pull request #8701: URL: https://github.com/apache/airflow/pull/8701#discussion_r426907487 ## File path: airflow/providers/amazon/aws/hooks/elasticache.py ## @@ -0,0 +1,259 @@ +# +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] VijayantSoni commented on a change in pull request #8701: Adding ElastiCache Hook for creating, describing and deleting replication groups

2020-05-18 Thread GitBox
VijayantSoni commented on a change in pull request #8701: URL: https://github.com/apache/airflow/pull/8701#discussion_r426907487 ## File path: airflow/providers/amazon/aws/hooks/elasticache.py ## @@ -0,0 +1,259 @@ +# +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] jaketf commented on issue #8673: Data Fusion Hook Start pipeline will succeed before pipeline is in RUNNING state

2020-05-18 Thread GitBox
jaketf commented on issue #8673: URL: https://github.com/apache/airflow/issues/8673#issuecomment-630444758 just an update it will be 2-3 weeks before anyone on my team has bandwidth for this. This is an automated message

[GitHub] [airflow] turbaszek commented on issue #8903: BigQueryHook refactor + deterministic BQ Job ID

2020-05-18 Thread GitBox
turbaszek commented on issue #8903: URL: https://github.com/apache/airflow/issues/8903#issuecomment-630443552 Hi @albertocalderari, I'm curently workining on refactor of BQ integration. I decided to abandon custom "run" and use `insert_job` method which will accept `job_id`:

[GitHub] [airflow] dimberman edited a comment on pull request #8904: Move k8s files out of contrib

2020-05-18 Thread GitBox
dimberman edited a comment on pull request #8904: URL: https://github.com/apache/airflow/pull/8904#issuecomment-630442531 @potiuk do you have any idea why all of these unrelated flake tests would fail? ``` tests/contrib/utils/base_gcp_system_test_case.py:89:0: F523

[GitHub] [airflow] dimberman commented on pull request #8904: Move k8s files out of contrib

2020-05-18 Thread GitBox
dimberman commented on pull request #8904: URL: https://github.com/apache/airflow/pull/8904#issuecomment-630442531 @potiuk do you have any idea why all of these unrelated flake tests would fail? This is an automated message

[GitHub] [airflow] ashb commented on a change in pull request #8904: Move k8s files out of contrib

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8904: URL: https://github.com/apache/airflow/pull/8904#discussion_r426896828 ## File path: .github/workflows/ci.yml ## @@ -123,7 +123,7 @@ jobs: run: ./scripts/ci/ci_free_space_on_ci.sh - name: "Build CI image ${{

[GitHub] [airflow] dimberman commented on pull request #8904: Move k8s files out of contrib

2020-05-18 Thread GitBox
dimberman commented on pull request #8904: URL: https://github.com/apache/airflow/pull/8904#issuecomment-630438161 @ashb some point tomorrow I might need some help on this PR. I'm getting a bunch of flake8 failures that seem completely unrelated to any code changes I made so not sure if

[GitHub] [airflow] dimberman commented on a change in pull request #8904: Move k8s files out of contrib

2020-05-18 Thread GitBox
dimberman commented on a change in pull request #8904: URL: https://github.com/apache/airflow/pull/8904#discussion_r426895969 ## File path: .github/workflows/ci.yml ## @@ -123,7 +123,7 @@ jobs: run: ./scripts/ci/ci_free_space_on_ci.sh - name: "Build CI image

[GitHub] [airflow] ashb commented on a change in pull request #8904: Move k8s files out of contrib

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8904: URL: https://github.com/apache/airflow/pull/8904#discussion_r426895275 ## File path: .github/workflows/ci.yml ## @@ -123,7 +123,7 @@ jobs: run: ./scripts/ci/ci_free_space_on_ci.sh - name: "Build CI image ${{

[GitHub] [airflow] dimberman commented on pull request #8904: Move k8s files out of contrib

2020-05-18 Thread GitBox
dimberman commented on pull request #8904: URL: https://github.com/apache/airflow/pull/8904#issuecomment-630396516 Many k8soperator/k8sexecutor PRs need to be essentially recreated from scratch because git can no longer merge. I'm hoping to make a few PRs like this to bring 1-10 and

[GitHub] [airflow] dimberman opened a new pull request #8904: Move k8s files out of contrib

2020-05-18 Thread GitBox
dimberman opened a new pull request #8904: URL: https://github.com/apache/airflow/pull/8904 --- Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Unit tests coverage for changes (not needed for documentation

[GitHub] [airflow] potiuk commented on pull request #8833: Azure storage 0.37.0 is not installable any more

2020-05-18 Thread GitBox
potiuk commented on pull request #8833: URL: https://github.com/apache/airflow/pull/8833#issuecomment-630382411 It's fixed in master If you try to install released 1.10.10 version in a stable way you should follow the

[GitHub] [airflow] potiuk commented on pull request #8898: Correct generation and package generation scripts

2020-05-18 Thread GitBox
potiuk commented on pull request #8898: URL: https://github.com/apache/airflow/pull/8898#issuecomment-630361086 * Wheels are now only -py3 python type * We have now two options: ``` --version-suffix-for-svn flag to generate rc candidate packages to upload to SVN or

[GitHub] [airflow] potiuk commented on pull request #8898: Correct generation and package generation scripts

2020-05-18 Thread GitBox
potiuk commented on pull request #8898: URL: https://github.com/apache/airflow/pull/8898#issuecomment-630352556 Hey @ashb @kaxil - I think I corrected all the issues that were mentioned in rc1 thread :). Please take a look :)

[GitHub] [airflow] Sinsin1367 commented on pull request #8734: Added optional logging for pod container statuses once a pod fails. T…

2020-05-18 Thread GitBox
Sinsin1367 commented on pull request #8734: URL: https://github.com/apache/airflow/pull/8734#issuecomment-630346426 @ashb @chrismclennon Do you mind taking a quick look and approving this if no more comments? This is an

[GitHub] [airflow] joppevos commented on pull request #8808: Support YAML input for CloudBuildCreateOperator

2020-05-18 Thread GitBox
joppevos commented on pull request #8808: URL: https://github.com/apache/airflow/pull/8808#issuecomment-630329569 @mik-laj The absolute path did indeed the trick. Build is passing now :thumbsup: This is an automated message

[GitHub] [airflow] jblakeman edited a comment on issue #8484: Airflow 1.10.7+ suppresses Operator logs

2020-05-18 Thread GitBox
jblakeman edited a comment on issue #8484: URL: https://github.com/apache/airflow/issues/8484#issuecomment-630315582 Had a chance to skim through #6627 and I could be mistaken but it seems like using `os._exit` in the forked child process without first flushing stdio buffers could be a

[GitHub] [airflow] jblakeman commented on issue #8484: Airflow 1.10.7+ suppresses Operator logs

2020-05-18 Thread GitBox
jblakeman commented on issue #8484: URL: https://github.com/apache/airflow/issues/8484#issuecomment-630315582 Had a chance to look through #6627 and I could be mistaken but it seems like using `os._exit` in the forked child process without first flushing stdio buffers could be a factor

[GitHub] [airflow] crazy-2020 commented on a change in pull request #8824: Make 'time_to_sleep_in_seconds' in dlp.py configurable

2020-05-18 Thread GitBox
crazy-2020 commented on a change in pull request #8824: URL: https://github.com/apache/airflow/pull/8824#discussion_r426760551 ## File path: airflow/providers/google/cloud/hooks/dlp.py ## @@ -207,6 +206,9 @@ def create_dlp_job( until it is set to DONE.

[GitHub] [airflow] crazy-2020 commented on a change in pull request #8824: Make 'time_to_sleep_in_seconds' in dlp.py configurable

2020-05-18 Thread GitBox
crazy-2020 commented on a change in pull request #8824: URL: https://github.com/apache/airflow/pull/8824#discussion_r426758286 ## File path: airflow/providers/google/cloud/hooks/dlp.py ## @@ -180,6 +178,7 @@ def create_dlp_job( timeout: Optional[float] = None,

[GitHub] [airflow] crazy-2020 commented on a change in pull request #8824: Make 'time_to_sleep_in_seconds' in dlp.py configurable

2020-05-18 Thread GitBox
crazy-2020 commented on a change in pull request #8824: URL: https://github.com/apache/airflow/pull/8824#discussion_r426757445 ## File path: airflow/providers/google/cloud/hooks/dlp.py ## @@ -207,6 +206,9 @@ def create_dlp_job( until it is set to DONE.

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8903: BigQueryHook refactor + deterministic BQ Job ID

2020-05-18 Thread GitBox
boring-cyborg[bot] commented on issue #8903: URL: https://github.com/apache/airflow/issues/8903#issuecomment-630288970 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] albertocalderari opened a new issue #8903: BigQueryHook refactor + deterministic BQ Job ID

2020-05-18 Thread GitBox
albertocalderari opened a new issue #8903: URL: https://github.com/apache/airflow/issues/8903 **Description** Looking at the code it seems like a lot of the logic in the BQ Hook is already implemented in the google API python library. This includes job polling, a nicer way

[GitHub] [airflow] randr97 commented on a change in pull request #8895: Add Delete/Create S3 bucket operators

2020-05-18 Thread GitBox
randr97 commented on a change in pull request #8895: URL: https://github.com/apache/airflow/pull/8895#discussion_r426706026 ## File path: airflow/providers/amazon/aws/hooks/s3.py ## @@ -664,6 +664,23 @@ def copy_object(self,

[GitHub] [airflow] felipefb commented on pull request #8833: Azure storage 0.37.0 is not installable any more

2020-05-18 Thread GitBox
felipefb commented on pull request #8833: URL: https://github.com/apache/airflow/pull/8833#issuecomment-630246885 I tried to install the azure package now and had a similar error. Is the new version already available? ``` Collecting azure-storage>=0.34.0; extra == "azure"

[GitHub] [airflow] realradical commented on pull request #8902: [AIRFLOW-8902] fix Dag Run UI execution date with timezone cannot be saved issue

2020-05-18 Thread GitBox
realradical commented on pull request #8902: URL: https://github.com/apache/airflow/pull/8902#issuecomment-630230910 @ashb sorry, accidentally closed https://github.com/apache/airflow/pull/8875 added the logic to parse datetime without TZ, and more test cases, please check if it is

[GitHub] [airflow] realradical opened a new pull request #8902: [AIRFLOW-8875] fix Dag Run UI execution date with timezone cannot be saved issue

2020-05-18 Thread GitBox
realradical opened a new pull request #8902: URL: https://github.com/apache/airflow/pull/8902 This PR is to address the following bug: https://github.com/apache/airflow/issues/8842 Airflow 1.10.10 displays datetime with timezone on the web UI. Currently, when user tries to create a dag

[GitHub] [airflow] realradical commented on pull request #8875: [AIRFLOW-8875] fix Dag Run UI execution date with timezone cannot be saved issue

2020-05-18 Thread GitBox
realradical commented on pull request #8875: URL: https://github.com/apache/airflow/pull/8875#issuecomment-630209425 @ashb i accidentally closed the pr. will create another one now. sorry about it This is an automated

[GitHub] [airflow] JavierLopezT opened a new pull request #8901: Secrets backend

2020-05-18 Thread GitBox
JavierLopezT opened a new pull request #8901: URL: https://github.com/apache/airflow/pull/8901 --- Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Unit tests coverage for changes (not needed for

[GitHub] [airflow] AmarEL opened a new issue #8900: Info about the parameter "conf" when triggering a DAG

2020-05-18 Thread GitBox
AmarEL opened a new issue #8900: URL: https://github.com/apache/airflow/issues/8900 **Description** The information and examples about the optional JSON `conf` when triggering a dag from UI or by CLI needs to be included in the documentation. **Use case / motivation**

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8900: Info about the parameter "conf" when triggering a DAG

2020-05-18 Thread GitBox
boring-cyborg[bot] commented on issue #8900: URL: https://github.com/apache/airflow/issues/8900#issuecomment-630186616 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] chamcca commented on a change in pull request #8749: add AWS StepFunctions integrations to the aws provider

2020-05-18 Thread GitBox
chamcca commented on a change in pull request #8749: URL: https://github.com/apache/airflow/pull/8749#discussion_r426622398 ## File path: airflow/providers/amazon/aws/operators/step_function_start_execution.py ## @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation

[GitHub] [airflow] chamcca commented on a change in pull request #8749: add AWS StepFunctions integrations to the aws provider

2020-05-18 Thread GitBox
chamcca commented on a change in pull request #8749: URL: https://github.com/apache/airflow/pull/8749#discussion_r426622233 ## File path: airflow/providers/amazon/aws/operators/step_function_get_execution_output.py ## @@ -0,0 +1,61 @@ +# Licensed to the Apache Software

[GitHub] [airflow] josh-gree commented on pull request #7954: Pinning max pandas version to 2.0 (lesser than) to allow pandas 1.0.

2020-05-18 Thread GitBox
josh-gree commented on pull request #7954: URL: https://github.com/apache/airflow/pull/7954#issuecomment-630178641 Hey thanks for this PR great to be able to have pandas > 1.0! Can I ask when this will make its way into a release?

[jira] [Commented] (AIRFLOW-6981) Move Google Cloud Build from Discovery API to Python Library

2020-05-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110271#comment-17110271 ] ASF GitHub Bot commented on AIRFLOW-6981: - ryanyuan commented on a change in pull request

[GitHub] [airflow] ryanyuan commented on a change in pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library

2020-05-18 Thread GitBox
ryanyuan commented on a change in pull request #8575: URL: https://github.com/apache/airflow/pull/8575#discussion_r426618675 ## File path: airflow/providers/google/cloud/operators/cloud_build.py ## @@ -15,186 +15,678 @@ # KIND, either express or implied. See the License for

[GitHub] [airflow] khyurri commented on a change in pull request #8818: WIP: Add support for Microsoft Azure Blob Storage in Google Cloud Storage Transfer Service operators

2020-05-18 Thread GitBox
khyurri commented on a change in pull request #8818: URL: https://github.com/apache/airflow/pull/8818#discussion_r426614269 ## File path: airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py ## @@ -97,8 +97,15 @@ class GcpTransferOperationStatus:

[GitHub] [airflow] potiuk commented on a change in pull request #8836: WIP - Add 3.8 to the test matrices.

2020-05-18 Thread GitBox
potiuk commented on a change in pull request #8836: URL: https://github.com/apache/airflow/pull/8836#discussion_r426609122 ## File path: requirements/requirements-python3.8.txt ## @@ -0,0 +1,370 @@ +# Editable install with no version control (apache-airflow==2.0.0.dev0)

[GitHub] [airflow] ashb commented on a change in pull request #8836: WIP - Add 3.8 to the test matrices.

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8836: URL: https://github.com/apache/airflow/pull/8836#discussion_r426592132 ## File path: requirements/requirements-python3.8.txt ## @@ -0,0 +1,370 @@ +# Editable install with no version control (apache-airflow==2.0.0.dev0) Review

[GitHub] [airflow] ashb commented on a change in pull request #8836: WIP - Add 3.8 to the test matrices.

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8836: URL: https://github.com/apache/airflow/pull/8836#discussion_r426591912 ## File path: requirements/requirements-python3.8.txt ## @@ -0,0 +1,370 @@ +# Editable install with no version control (apache-airflow==2.0.0.dev0) Review

[GitHub] [airflow] randr97 commented on a change in pull request #8895: Add Delete/Create S3 bucket operators

2020-05-18 Thread GitBox
randr97 commented on a change in pull request #8895: URL: https://github.com/apache/airflow/pull/8895#discussion_r426588128 ## File path: airflow/providers/amazon/aws/operators/s3_bucket.py ## @@ -0,0 +1,132 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] randr97 commented on a change in pull request #8895: Add Delete/Create S3 bucket operators

2020-05-18 Thread GitBox
randr97 commented on a change in pull request #8895: URL: https://github.com/apache/airflow/pull/8895#discussion_r426587315 ## File path: airflow/providers/amazon/aws/operators/s3_bucket.py ## @@ -0,0 +1,132 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] ashb commented on a change in pull request #8701: Adding ElastiCache Hook for creating, describing and deleting replication groups

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8701: URL: https://github.com/apache/airflow/pull/8701#discussion_r426578726 ## File path: airflow/providers/amazon/aws/hooks/elasticache.py ## @@ -0,0 +1,259 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[GitHub] [airflow] ashb commented on a change in pull request #8701: Adding ElastiCache Hook for creating, describing and deleting replication groups

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8701: URL: https://github.com/apache/airflow/pull/8701#discussion_r426578034 ## File path: airflow/providers/amazon/aws/hooks/elasticache.py ## @@ -0,0 +1,259 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[GitHub] [airflow] ashb commented on a change in pull request #8701: Adding ElastiCache Hook for creating, describing and deleting replication groups

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8701: URL: https://github.com/apache/airflow/pull/8701#discussion_r426577707 ## File path: airflow/providers/amazon/aws/hooks/elasticache.py ## @@ -0,0 +1,259 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[GitHub] [airflow] ashb commented on a change in pull request #8701: Adding ElastiCache Hook for creating, describing and deleting replication groups

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8701: URL: https://github.com/apache/airflow/pull/8701#discussion_r426577191 ## File path: airflow/providers/amazon/aws/hooks/elasticache.py ## @@ -0,0 +1,259 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[GitHub] [airflow] ashb commented on a change in pull request #8701: Adding ElastiCache Hook for creating, describing and deleting replication groups

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8701: URL: https://github.com/apache/airflow/pull/8701#discussion_r426576775 ## File path: airflow/providers/amazon/aws/hooks/elasticache.py ## @@ -0,0 +1,259 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[GitHub] [airflow] ashb commented on a change in pull request #8701: Adding ElastiCache Hook for creating, describing and deleting replication groups

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8701: URL: https://github.com/apache/airflow/pull/8701#discussion_r426576297 ## File path: airflow/providers/amazon/aws/hooks/elasticache.py ## @@ -0,0 +1,259 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[GitHub] [airflow] ashb commented on a change in pull request #8701: Adding ElastiCache Hook for creating, describing and deleting replication groups

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8701: URL: https://github.com/apache/airflow/pull/8701#discussion_r426575633 ## File path: airflow/providers/amazon/aws/hooks/elasticache.py ## @@ -0,0 +1,259 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[GitHub] [airflow] ashb commented on a change in pull request #8701: Adding ElastiCache Hook for creating, describing and deleting replication groups

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8701: URL: https://github.com/apache/airflow/pull/8701#discussion_r426575179 ## File path: airflow/providers/amazon/aws/hooks/elasticache.py ## @@ -0,0 +1,259 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[GitHub] [airflow] ashb commented on a change in pull request #8895: Add Delete/Create S3 bucket operators

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8895: URL: https://github.com/apache/airflow/pull/8895#discussion_r426569097 ## File path: airflow/providers/amazon/aws/hooks/s3.py ## @@ -664,6 +664,23 @@ def copy_object(self,

[GitHub] [airflow] ashb commented on a change in pull request #8824: Make 'time_to_sleep_in_seconds' in dlp.py configurable

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8824: URL: https://github.com/apache/airflow/pull/8824#discussion_r426567978 ## File path: airflow/providers/google/cloud/hooks/dlp.py ## @@ -207,6 +206,9 @@ def create_dlp_job( until it is set to DONE. :type

[GitHub] [airflow] ashb commented on a change in pull request #8824: Make 'time_to_sleep_in_seconds' in dlp.py configurable

2020-05-18 Thread GitBox
ashb commented on a change in pull request #8824: URL: https://github.com/apache/airflow/pull/8824#discussion_r426567371 ## File path: airflow/providers/google/cloud/hooks/dlp.py ## @@ -207,6 +206,9 @@ def create_dlp_job( until it is set to DONE. :type

[GitHub] [airflow] ashb commented on pull request #8875: [AIRFLOW-8875] fix Dag Run UI execution date with timezone cannot be saved issue

2020-05-18 Thread GitBox
ashb commented on pull request #8875: URL: https://github.com/apache/airflow/pull/8875#issuecomment-630127673 @realradical I think the backend change you have here is good! With it it should accept any datetime that the UI displays, but without it it would only accept TZ in the current

[GitHub] [airflow] jeffolsi closed issue #8565: SalesforceHook missing method to return dataframe

2020-05-18 Thread GitBox
jeffolsi closed issue #8565: URL: https://github.com/apache/airflow/issues/8565 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] potiuk commented on pull request #8898: Correct generation and package generation scripts

2020-05-18 Thread GitBox
potiuk commented on pull request #8898: URL: https://github.com/apache/airflow/pull/8898#issuecomment-630097988 > > @ashb @kaxil -> I figured out that we have really two modes of release - one for "SVN" where we only rename the files with rc but the version inside is without the suffix,

[GitHub] [airflow] ashb commented on pull request #8898: Correct generation and package generation scripts

2020-05-18 Thread GitBox
ashb commented on pull request #8898: URL: https://github.com/apache/airflow/pull/8898#issuecomment-630095741 > @ashb @kaxil -> I figured out that we have really two modes of release - one for "SVN" where we only rename the files with rc but the version inside is without the suffix, and

[GitHub] [airflow] feluelle commented on a change in pull request #8877: Add S3ToRedshift example dag and system test

2020-05-18 Thread GitBox
feluelle commented on a change in pull request #8877: URL: https://github.com/apache/airflow/pull/8877#discussion_r426519274 ## File path: tests/test_utils/postgres_system_helpers.py ## @@ -0,0 +1,97 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more

[GitHub] [airflow] realradical edited a comment on pull request #8875: [AIRFLOW-8875] fix Dag Run UI execution date with timezone cannot be saved issue

2020-05-18 Thread GitBox
realradical edited a comment on pull request #8875: URL: https://github.com/apache/airflow/pull/8875#issuecomment-630081091 > @conf_vars @ashb I tried changing the timezone in the UI and it works! I used `@conf_vars({("core", "default_timezone"): "America/Toronto"})` in the

[GitHub] [airflow] realradical closed pull request #8875: [AIRFLOW-8875] fix Dag Run UI execution date with timezone cannot be saved issue

2020-05-18 Thread GitBox
realradical closed pull request #8875: URL: https://github.com/apache/airflow/pull/8875 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] realradical commented on pull request #8875: [AIRFLOW-8875] fix Dag Run UI execution date with timezone cannot be saved issue

2020-05-18 Thread GitBox
realradical commented on pull request #8875: URL: https://github.com/apache/airflow/pull/8875#issuecomment-630081091 > @conf_vars @ashb This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] JavierLTPromofarma opened a new issue #8899: Get UI color of DAGs from airflow.cfg

2020-05-18 Thread GitBox
JavierLTPromofarma opened a new issue #8899: URL: https://github.com/apache/airflow/issues/8899 **Description** Being able to get different ui colors than the default ones for operators from airflow.cfg file **Use case / motivation** I want to have more vivid colors for

  1   2   >