[GitHub] [airflow] RyanSiu1995 commented on issue #8780: Missing logs if the pod was killed by Kubernetes

2020-05-08 Thread GitBox
RyanSiu1995 commented on issue #8780: URL: https://github.com/apache/airflow/issues/8780#issuecomment-625756560 @msumit Thank you for your reply. I am thinking a change like this. We reinitiate the database session dynamically in this util files.

[GitHub] [airflow] RyanSiu1995 commented on issue #8780: Missing logs if the pod was killed by Kubernetes

2020-05-08 Thread GitBox
RyanSiu1995 commented on issue #8780: URL: https://github.com/apache/airflow/issues/8780#issuecomment-625761275 But, to be honest, it is pretty odd that the function `create_session()` will not actually have any action of creation. It just tries to return the session object in

[GitHub] [airflow] potiuk opened a new issue #8785: Move out all unnecessary binaries installed in CI image

2020-05-08 Thread GitBox
potiuk opened a new issue #8785: URL: https://github.com/apache/airflow/issues/8785 **Description** We have a lot of unnecessary binaries/libraries baked into the CI image. And we can move it out in the form of integration images downloaded separately by docker-compose.

[GitHub] [airflow] RyanSiu1995 commented on issue #8780: Missing logs if the pod was killed by Kubernetes

2020-05-08 Thread GitBox
RyanSiu1995 commented on issue #8780: URL: https://github.com/apache/airflow/issues/8780#issuecomment-625769729 Yes, that's exactly what happen with my case. This is an automated message from the Apache Git Service. To

[GitHub] [airflow] boittega opened a new pull request #8786: Add WorldRemit as Airflow user

2020-05-08 Thread GitBox
boittega opened a new pull request #8786: URL: https://github.com/apache/airflow/pull/8786 Adding WorldRemit as Airflow user. --- Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Unit tests coverage

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102530#comment-17102530 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on pull request #6007: URL:

[GitHub] [airflow] kaxil commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-625791176 Static check are failing: ``` diff --git a/docs/operators-and-hooks-ref.rst b/docs/operators-and-hooks-ref.rst index 0fcf11b..0b024ff 100644 ---

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102544#comment-17102544 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on pull request #6007: URL:

[GitHub] [airflow] kaxil commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-625799603 Another failed test ``` * Module airflow.providers.amazon.aws.hooks.glue airflow/providers/amazon/aws/hooks/glue.py:70:8: C0103: Attribute name "S3_GLUE_LOGS"

[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102549#comment-17102549 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request

[GitHub] [airflow] TeddyHartanto commented on a change in pull request #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream

2020-05-08 Thread GitBox
TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r422125930 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG

[GitHub] [airflow] Rcharriol commented on pull request #7777: [AIRFLOW-Date_range compatible with schedule_interval]

2020-05-08 Thread GitBox
Rcharriol commented on pull request #: URL: https://github.com/apache/airflow/pull/#issuecomment-625800899 @mik-laj Im having troubles to pass the tests, and got no clue why is failing, can i get some help please?

[GitHub] [airflow] Blacksmoke16 commented on pull request #8737: Add note that the RBAC UI is required for extra links

2020-05-08 Thread GitBox
Blacksmoke16 commented on pull request #8737: URL: https://github.com/apache/airflow/pull/8737#issuecomment-625802179 Oops, I borked the merge @ashb. PR is recreated as https://github.com/apache/airflow/pull/8788. This is

[GitHub] [airflow] kaxil commented on pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
kaxil commented on pull request #8772: URL: https://github.com/apache/airflow/pull/8772#issuecomment-625803983 > Curious failure: > > ``` > E AssertionError: default_args[start_date] matches > E assert == > ``` > > I wonder why I

[GitHub] [airflow] kaxil commented on a change in pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #8772: URL: https://github.com/apache/airflow/pull/8772#discussion_r422132180 ## File path: airflow/serialization/serialized_objects.py ## @@ -395,6 +411,59 @@ def deserialize_operator(cls, encoded_op: Dict[str, Any]) ->

[airflow] branch v1-10-stable updated: Add note extra links only render on when using RBAC webserver (#8788)

2020-05-08 Thread ash
This is an automated email from the ASF dual-hosted git repository. ash pushed a commit to branch v1-10-stable in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/v1-10-stable by this push: new b099571 Add note extra links only

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8788: Add note that the RBAC UI is required for extra links

2020-05-08 Thread GitBox
boring-cyborg[bot] commented on pull request #8788: URL: https://github.com/apache/airflow/pull/8788#issuecomment-625808396 Awesome work, congrats on your first merged pull request! This is an automated message from the

[GitHub] [airflow] ashb commented on pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
ashb commented on pull request #8772: URL: https://github.com/apache/airflow/pull/8772#issuecomment-625814879 Oh, cos it's parsing in a subprocess. H! I don't think we need to parse _all_ dags in a subprocess, just one is enough.

[GitHub] [airflow] nadflinn commented on a change in pull request #8695: Make celery worker_prefetch_multiplier configurable

2020-05-08 Thread GitBox
nadflinn commented on a change in pull request #8695: URL: https://github.com/apache/airflow/pull/8695#discussion_r422156337 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -550,6 +550,15 @@ worker_concurrency = 8 # Example: worker_autoscale = 16,12 #

[GitHub] [airflow] aneesh-joseph commented on a change in pull request #8777: Add Production Helm chart support

2020-05-08 Thread GitBox
aneesh-joseph commented on a change in pull request #8777: URL: https://github.com/apache/airflow/pull/8777#discussion_r422158586 ## File path: chart/templates/configmap.yaml ## @@ -0,0 +1,108 @@ + +## Airflow ConfigMap

[GitHub] [airflow] kaxil commented on a change in pull request #8695: Make celery worker_prefetch_multiplier configurable

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #8695: URL: https://github.com/apache/airflow/pull/8695#discussion_r422158345 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -550,6 +550,15 @@ worker_concurrency = 8 # Example: worker_autoscale = 16,12 #

[GitHub] [airflow] aneesh-joseph commented on a change in pull request #8777: Add Production Helm chart support

2020-05-08 Thread GitBox
aneesh-joseph commented on a change in pull request #8777: URL: https://github.com/apache/airflow/pull/8777#discussion_r422158586 ## File path: chart/templates/configmap.yaml ## @@ -0,0 +1,108 @@ + +## Airflow ConfigMap

[GitHub] [airflow] aneesh-joseph commented on a change in pull request #8777: Add Production Helm chart support

2020-05-08 Thread GitBox
aneesh-joseph commented on a change in pull request #8777: URL: https://github.com/apache/airflow/pull/8777#discussion_r422158586 ## File path: chart/templates/configmap.yaml ## @@ -0,0 +1,108 @@ + +## Airflow ConfigMap

[GitHub] [airflow] aneesh-joseph commented on a change in pull request #8777: Add Production Helm chart support

2020-05-08 Thread GitBox
aneesh-joseph commented on a change in pull request #8777: URL: https://github.com/apache/airflow/pull/8777#discussion_r422158586 ## File path: chart/templates/configmap.yaml ## @@ -0,0 +1,108 @@ + +## Airflow ConfigMap

[GitHub] [airflow] aneesh-joseph commented on a change in pull request #8777: Add Production Helm chart support

2020-05-08 Thread GitBox
aneesh-joseph commented on a change in pull request #8777: URL: https://github.com/apache/airflow/pull/8777#discussion_r422158586 ## File path: chart/templates/configmap.yaml ## @@ -0,0 +1,108 @@ + +## Airflow ConfigMap

[GitHub] [airflow] kaxil commented on a change in pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #6007: URL: https://github.com/apache/airflow/pull/6007#discussion_r422124800 ## File path: tests/providers/amazon/aws/hooks/test_aws_glue_job_hook.py ## @@ -0,0 +1,107 @@ +# +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] Blacksmoke16 opened a new pull request #8788: Add note that the RBAC UI is required for extra links

2020-05-08 Thread GitBox
Blacksmoke16 opened a new pull request #8788: URL: https://github.com/apache/airflow/pull/8788 After trying unsuccessfully to add extra links based on the [guide](https://airflow.apache.org/docs/stable/howto/define_extra_link.html), I came across some stackoverflow posts,

[GitHub] [airflow] ashb commented on pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
ashb commented on pull request #8772: URL: https://github.com/apache/airflow/pull/8772#issuecomment-625803075 Curious failure: ``` E AssertionError: default_args[start_date] matches E assert == ``` I wonder why I didn't see that

[GitHub] [airflow] kaxil commented on a change in pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #8772: URL: https://github.com/apache/airflow/pull/8772#discussion_r422127805 ## File path: tests/serialization/test_dag_serialization.py ## @@ -273,58 +271,70 @@ def test_deserialization(self): for dag_id in

[jira] [Commented] (AIRFLOW-3369) Un-pausing a DAG with catchup =False creates an extra DAG run (1.10)

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102581#comment-17102581 ] ASF GitHub Bot commented on AIRFLOW-3369: - ashb commented on pull request #8776: URL:

[GitHub] [airflow] ashb commented on pull request #8776: WIP: [AIRFLOW-3369] BugFix: Unpausing a DAG with catchup=False creates an extra DAG run

2020-05-08 Thread GitBox
ashb commented on pull request #8776: URL: https://github.com/apache/airflow/pull/8776#issuecomment-625811055 Lol, one line fixes are the Best* Tests please :) (I'm sure you'd get around to it) \* The worst.

[GitHub] [airflow] ashb commented on pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
ashb commented on pull request #8772: URL: https://github.com/apache/airflow/pull/8772#issuecomment-625814458 Curious, the DAG with the problem has `'start_date': datetime.utcnow()` -- that's "wrong" but I wouldn't have expected this test to fail. I guess we are loading the dag twice,

[GitHub] [airflow] nadflinn commented on a change in pull request #8695: Make celery worker_prefetch_multiplier configurable

2020-05-08 Thread GitBox
nadflinn commented on a change in pull request #8695: URL: https://github.com/apache/airflow/pull/8695#discussion_r422154400 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -550,6 +550,15 @@ worker_concurrency = 8 # Example: worker_autoscale = 16,12 #

[GitHub] [airflow] aneesh-joseph commented on a change in pull request #8777: Add Production Helm chart support

2020-05-08 Thread GitBox
aneesh-joseph commented on a change in pull request #8777: URL: https://github.com/apache/airflow/pull/8777#discussion_r422158586 ## File path: chart/templates/configmap.yaml ## @@ -0,0 +1,108 @@ + +## Airflow ConfigMap

[GitHub] [airflow] kaxil commented on a change in pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #6007: URL: https://github.com/apache/airflow/pull/6007#discussion_r422125128 ## File path: tests/providers/amazon/aws/hooks/test_aws_glue_job_hook.py ## @@ -0,0 +1,107 @@ +# +# Licensed to the Apache Software Foundation (ASF) under

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102546#comment-17102546 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on a change in pull request #6007:

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102545#comment-17102545 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on a change in pull request #6007:

[GitHub] [airflow] nadflinn commented on a change in pull request #8695: Make celery worker_prefetch_multiplier configurable

2020-05-08 Thread GitBox
nadflinn commented on a change in pull request #8695: URL: https://github.com/apache/airflow/pull/8695#discussion_r422152685 ## File path: airflow/config_templates/config.yml ## @@ -1130,6 +1130,19 @@ type: string example: 16,12 default: ~ +- name:

[GitHub] [airflow] ashb commented on a change in pull request #8734: Added optional logging for pod container statuses once a pod fails. T…

2020-05-08 Thread GitBox
ashb commented on a change in pull request #8734: URL: https://github.com/apache/airflow/pull/8734#discussion_r421989967 ## File path: airflow/kubernetes/pod_launcher.py ## @@ -183,6 +183,23 @@ def read_pod_logs(self, pod: V1Pod, tail_lines: int = 10): 'There

[GitHub] [airflow] turbaszek commented on issue #8703: Support for set in XCom serialization

2020-05-08 Thread GitBox
turbaszek commented on issue #8703: URL: https://github.com/apache/airflow/issues/8703#issuecomment-625700758 > XCom used to be pickle by default (and still is on 1.10?), that'll be why it has gone un-noticed for so long. Imho the main reason were side effects in tests. Otherwise,

[GitHub] [airflow] mik-laj commented on issue #8207: Create guide for Machine Learning Engine operators

2020-05-08 Thread GitBox
mik-laj commented on issue #8207: URL: https://github.com/apache/airflow/issues/8207#issuecomment-625723473 @U-Ozdemir We try to use a logical clock instead of a real clock in the community. The order of operations is more important than the time the task was completed. Some tasks need

[GitHub] [airflow] mik-laj commented on pull request #8768: [POC] Mark keywords-only arguments in hook method signatures

2020-05-08 Thread GitBox
mik-laj commented on pull request #8768: URL: https://github.com/apache/airflow/pull/8768#issuecomment-625730448 A similar case is apply_defaults decorator and any operator constructor. This is an automated message from the

[GitHub] [airflow] msumit commented on issue #8780: Missing logs if the pod was killed by Kubernetes

2020-05-08 Thread GitBox
msumit commented on issue #8780: URL: https://github.com/apache/airflow/issues/8780#issuecomment-625748213 AFAIK the tasks logs are uploaded to remote storage once the task is completed, be it Kubernetes, Celery, or even Local executors. One simple solution is to make use of multi

[GitHub] [airflow] turbaszek commented on pull request #8227: Add run_type to DagRun

2020-05-08 Thread GitBox
turbaszek commented on pull request #8227: URL: https://github.com/apache/airflow/pull/8227#issuecomment-625757619 > https://github.com/apache/airflow/blob/master/airflow/operators/dagrun_operator.py#L65-L71 @mik-laj do you think we should add new run type like `TRIGGERED`?

[GitHub] [airflow] msumit commented on issue #8780: Missing logs if the pod was killed by Kubernetes

2020-05-08 Thread GitBox
msumit commented on issue #8780: URL: https://github.com/apache/airflow/issues/8780#issuecomment-625768894 @RyanSiu1995 so it means that the worker pod is trying to upload the logs to GCS when being killed, but failing due to some issue in getting DB session?

[GitHub] [airflow] kaxil commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-625783624 Docs are failing: ``` The HTML pages are in _build/html. Error 1 WARNING: Field list ends without a blank line; unexpected

[GitHub] [airflow] mik-laj commented on pull request #8721: Add OpenAPI specification (II)

2020-05-08 Thread GitBox
mik-laj commented on pull request #8721: URL: https://github.com/apache/airflow/pull/8721#issuecomment-625789285 Preview: https://editor.swagger.io/?url=https://raw.githubusercontent.com/PolideaInternal/airflow/openapi-spec/openapi.yaml

[GitHub] [airflow] kaxil commented on a change in pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #6007: URL: https://github.com/apache/airflow/pull/6007#discussion_r422115507 ## File path: docs/operators-and-hooks-ref.rst ## @@ -329,6 +329,12 @@ These integrations allow you to perform various operations within the Amazon Web

[GitHub] [airflow] mshalak-nix commented on issue #8760: Webserver becomes unusable after 100,000 tasks were completed

2020-05-08 Thread GitBox
mshalak-nix commented on issue #8760: URL: https://github.com/apache/airflow/issues/8760#issuecomment-625792761 Looks like I figured it out. The issue was with prometheus exporter I used, here is the fix: https://github.com/robinhood/airflow-prometheus-exporter/pull/25 It generated

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102534#comment-17102534 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on a change in pull request #6007:

[GitHub] [airflow] boittega opened a new pull request #8787: [8723] Spark JDBC Hook conf default parameter

2020-05-08 Thread GitBox
boittega opened a new pull request #8787: URL: https://github.com/apache/airflow/pull/8787 When spark_conf parameter is not set, the SparkJdbcHook raises an error because it is not iterable. Following the same behaviour than SparkSubmitHook. Resolves #8723 --- Make sure to

[GitHub] [airflow] kaxil commented on a change in pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #8772: URL: https://github.com/apache/airflow/pull/8772#discussion_r422119825 ## File path: airflow/serialization/serialized_objects.py ## @@ -119,11 +120,16 @@ def _is_primitive(cls, var: Any) -> bool: @classmethod def

[GitHub] [airflow] kaxil commented on a change in pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #8772: URL: https://github.com/apache/airflow/pull/8772#discussion_r422121186 ## File path: airflow/serialization/serialized_objects.py ## @@ -285,10 +297,14 @@ class SerializedBaseOperator(BaseOperator, BaseSerialization):

[GitHub] [airflow] elukey commented on pull request #5659: [AIRFLOW-5033] Switched to snakebite-py3

2020-05-08 Thread GitBox
elukey commented on pull request #5659: URL: https://github.com/apache/airflow/pull/5659#issuecomment-625749508 Good news: I got unblocked, I was able to make snakebite working with `sasl3` after a lot of time spent between swearing and crying :D, details in

[jira] [Commented] (AIRFLOW-5033) Cannot install apache-airflow[kerberos] on Python 3

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102445#comment-17102445 ] ASF GitHub Bot commented on AIRFLOW-5033: - elukey commented on pull request #5659: URL:

[GitHub] [airflow] potiuk opened a new issue #8783: Move Hadoop distro out of CI image

2020-05-08 Thread GitBox
potiuk opened a new issue #8783: URL: https://github.com/apache/airflow/issues/8783 **Description** Hadoop distro is unnecessarily part of the CI image. It should be moved out and used separately **Use case / motivation** Download and install Hadoop image only when

[jira] [Commented] (AIRFLOW-7068) Create EC2 Hook, Operator and Sensor

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102485#comment-17102485 ] ASF GitHub Bot commented on AIRFLOW-7068: - mustafagok commented on pull request #7731: URL:

[GitHub] [airflow] mustafagok commented on pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor

2020-05-08 Thread GitBox
mustafagok commented on pull request #7731: URL: https://github.com/apache/airflow/pull/7731#issuecomment-625770673 > I restarted the failed travis check. Lets wait until it succeds and then I can merge it. @mustafagok can you add example dag + system test in another PR? That would be

[GitHub] [airflow] potiuk commented on issue #8774: Move singularity out of main CI tests and into separate docker image/system test

2020-05-08 Thread GitBox
potiuk commented on issue #8774: URL: https://github.com/apache/airflow/issues/8774#issuecomment-625770630 Not sure if it is still needed. Maybe @vsoch can tell more about it whether it is actually needed in the image. I think we had a least few "integration" kind of tests where we run

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102519#comment-17102519 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on a change in pull request #6007:

[GitHub] [airflow] kaxil commented on a change in pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #6007: URL: https://github.com/apache/airflow/pull/6007#discussion_r422105893 ## File path: airflow/providers/amazon/aws/operators/glue.py ## @@ -0,0 +1,114 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102518#comment-17102518 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on pull request #6007: URL:

[GitHub] [airflow] potiuk opened a new issue #8782: Move Kubernetes tests and kind setup out of the container to host

2020-05-08 Thread GitBox
potiuk opened a new issue #8782: URL: https://github.com/apache/airflow/issues/8782 **Description** We should move kind setup out of the container for CI to the host environment **Use case / motivation** Makes CI image smaller and setup easier for local testing.

[GitHub] [airflow] potiuk opened a new issue #8784: Move Minicluster out of the main CI image

2020-05-08 Thread GitBox
potiuk opened a new issue #8784: URL: https://github.com/apache/airflow/issues/8784 **Description** Minicluster is unnecessarily part of the CI image. It should be moved out and used separately **Use case / motivation** Download and install Mincluster image only when

[GitHub] [airflow] potiuk commented on pull request #8768: [POC] Mark keywords-only arguments in hook method signatures

2020-05-08 Thread GitBox
potiuk commented on pull request #8768: URL: https://github.com/apache/airflow/pull/8768#issuecomment-625787883 However static checks are failing as our annotation is confusing pylint :( This is an automated message from

[GitHub] [airflow] mshalak-nix edited a comment on issue #8760: Webserver becomes unusable after 100,000 tasks were completed

2020-05-08 Thread GitBox
mshalak-nix edited a comment on issue #8760: URL: https://github.com/apache/airflow/issues/8760#issuecomment-625792761 Looks like I figured it out. The issue was with prometheus exporter I used, here is the fix: https://github.com/robinhood/airflow-prometheus-exporter/pull/25 It

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102538#comment-17102538 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on pull request #6007: URL:

[GitHub] [airflow] kaxil commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-625794242 I have added 2 commits to fix some of the issues ! This is an automated message from the Apache Git Service. To

[GitHub] [airflow] webster-chainalysis commented on issue #8212: Can't read S3 remote logs using Airflow 1.10.9

2020-05-08 Thread GitBox
webster-chainalysis commented on issue #8212: URL: https://github.com/apache/airflow/issues/8212#issuecomment-625893851 @dimon222 - Yes. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] dferguson992 commented on a change in pull request #7407: [AIRFLOW-6786] Add KafkaConsumerHook, KafkaProduerHook and KafkaSensor

2020-05-08 Thread GitBox
dferguson992 commented on a change in pull request #7407: URL: https://github.com/apache/airflow/pull/7407#discussion_r422236286 ## File path: airflow/providers/apache/kafka/hooks/kafka_consumer_hook.py ## @@ -0,0 +1,80 @@ +# +# Licensed to the Apache Software Foundation (ASF)

[jira] [Commented] (AIRFLOW-6786) Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102707#comment-17102707 ] ASF GitHub Bot commented on AIRFLOW-6786: - dferguson992 commented on a change in pull request

[GitHub] [airflow] dferguson992 commented on a change in pull request #7407: [AIRFLOW-6786] Add KafkaConsumerHook, KafkaProduerHook and KafkaSensor

2020-05-08 Thread GitBox
dferguson992 commented on a change in pull request #7407: URL: https://github.com/apache/airflow/pull/7407#discussion_r422240631 ## File path: airflow/providers/apache/kafka/sensors/kafka_sensor.py ## @@ -0,0 +1,76 @@ +# +# Licensed to the Apache Software Foundation (ASF)

[jira] [Commented] (AIRFLOW-6786) Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102715#comment-17102715 ] ASF GitHub Bot commented on AIRFLOW-6786: - dferguson992 commented on a change in pull request

[GitHub] [airflow] turbaszek commented on pull request #8610: Added Upload Multiple Entity Read Files to specified big query dataset

2020-05-08 Thread GitBox
turbaszek commented on pull request #8610: URL: https://github.com/apache/airflow/pull/8610#issuecomment-625898833 @michalslowikowski00 can you rebase please? This is an automated message from the Apache Git Service. To

[GitHub] [airflow] Sinsin1367 commented on a change in pull request #8734: Added optional logging for pod container statuses once a pod fails. T…

2020-05-08 Thread GitBox
Sinsin1367 commented on a change in pull request #8734: URL: https://github.com/apache/airflow/pull/8734#discussion_r422181509 ## File path: airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py ## @@ -177,6 +181,7 @@ def __init__(self, # pylint:

[GitHub] [airflow] anitakar commented on pull request #8769: Fix running DAG from UI when dag serialization is on

2020-05-08 Thread GitBox
anitakar commented on pull request #8769: URL: https://github.com/apache/airflow/pull/8769#issuecomment-625848154 @mik-laj Do you want to take a look? This is an automated message from the Apache Git Service. To respond to

[airflow] branch master updated: Add WorldRemit as Airflow user (#8786)

2020-05-08 Thread turbaszek
This is an automated email from the ASF dual-hosted git repository. turbaszek pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new 09770e4 Add WorldRemit as Airflow user

[GitHub] [airflow] Sinsin1367 commented on a change in pull request #8734: Added optional logging for pod container statuses once a pod fails. T…

2020-05-08 Thread GitBox
Sinsin1367 commented on a change in pull request #8734: URL: https://github.com/apache/airflow/pull/8734#discussion_r422179683 ## File path: airflow/kubernetes/pod_launcher.py ## @@ -183,6 +183,23 @@ def read_pod_logs(self, pod: V1Pod, tail_lines: int = 10):

[GitHub] [airflow] dsaiztc commented on a change in pull request #6523: [AIRFLOW-5873] KubernetesPodOperator fixes and test

2020-05-08 Thread GitBox
dsaiztc commented on a change in pull request #6523: URL: https://github.com/apache/airflow/pull/6523#discussion_r422286551 ## File path: airflow/contrib/operators/kubernetes_pod_operator.py ## @@ -158,17 +167,17 @@ def execute(self, context): raise

[jira] [Commented] (AIRFLOW-5873) KubernetesPodOperator fixes and test

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102797#comment-17102797 ] ASF GitHub Bot commented on AIRFLOW-5873: - dsaiztc commented on a change in pull request #6523:

[GitHub] [airflow] dsaiztc commented on a change in pull request #6523: [AIRFLOW-5873] KubernetesPodOperator fixes and test

2020-05-08 Thread GitBox
dsaiztc commented on a change in pull request #6523: URL: https://github.com/apache/airflow/pull/6523#discussion_r422286551 ## File path: airflow/contrib/operators/kubernetes_pod_operator.py ## @@ -158,17 +167,17 @@ def execute(self, context): raise

[jira] [Commented] (AIRFLOW-5873) KubernetesPodOperator fixes and test

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102801#comment-17102801 ] ASF GitHub Bot commented on AIRFLOW-5873: - dsaiztc commented on a change in pull request #6523:

[GitHub] [airflow] ddelange commented on a change in pull request #6523: [AIRFLOW-5873] KubernetesPodOperator fixes and test

2020-05-08 Thread GitBox
ddelange commented on a change in pull request #6523: URL: https://github.com/apache/airflow/pull/6523#discussion_r422295222 ## File path: airflow/contrib/operators/kubernetes_pod_operator.py ## @@ -158,17 +167,17 @@ def execute(self, context): raise

[jira] [Commented] (AIRFLOW-5873) KubernetesPodOperator fixes and test

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102812#comment-17102812 ] ASF GitHub Bot commented on AIRFLOW-5873: - ddelange commented on a change in pull request

[GitHub] [airflow] ddelange commented on a change in pull request #6523: [AIRFLOW-5873] KubernetesPodOperator fixes and test

2020-05-08 Thread GitBox
ddelange commented on a change in pull request #6523: URL: https://github.com/apache/airflow/pull/6523#discussion_r422295222 ## File path: airflow/contrib/operators/kubernetes_pod_operator.py ## @@ -158,17 +167,17 @@ def execute(self, context): raise

[jira] [Commented] (AIRFLOW-5873) KubernetesPodOperator fixes and test

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102820#comment-17102820 ] ASF GitHub Bot commented on AIRFLOW-5873: - ddelange commented on a change in pull request

[GitHub] [airflow] tooptoop4 opened a new issue #8789: Ability to throttle scheduler bursts on single pool

2020-05-08 Thread GitBox
tooptoop4 opened a new issue #8789: URL: https://github.com/apache/airflow/issues/8789 Current issue is I have pools configured on same type of task where I can have up to max 100 pool slots of that task type running in parallel BUT I want ability to say don't START running more than 5 at

[GitHub] [airflow] ashb commented on a change in pull request #8757: Fix the incorrect description of pod_mutation_hook in kubernetes.rst

2020-05-08 Thread GitBox
ashb commented on a change in pull request #8757: URL: https://github.com/apache/airflow/pull/8757#discussion_r422377934 ## File path: docs/kubernetes.rst ## @@ -40,11 +40,44 @@ that has the ability to mutate pod objects before sending them to the Kubernetes for scheduling.

[GitHub] [airflow] ashb commented on pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
ashb commented on pull request #8772: URL: https://github.com/apache/airflow/pull/8772#issuecomment-626013593 > Oh, cos it's parsing in a subprocess. H! > > I don't think we need to parse _all_ dags in a subprocess, just one is enough. Changed it to only parse

[GitHub] [airflow] ashb opened a new pull request #8790: Update example SingularityOperator dag

2020-05-08 Thread GitBox
ashb opened a new pull request #8790: URL: https://github.com/apache/airflow/pull/8790 The main thing I was fixing here was `start_date=utcnow()` which is always going to be wrong (discovered via a test in #8772). While I was updating the DAG I updated it to use context manager,

[GitHub] [airflow] ashb commented on pull request #8769: Fix running DAG from UI when dag serialization is on

2020-05-08 Thread GitBox
ashb commented on pull request #8769: URL: https://github.com/apache/airflow/pull/8769#issuecomment-626022797 What are the steps to reproduce this error with serialization turned on please? This is an automated message from

[GitHub] [airflow] ashb commented on a change in pull request #8757: Fix the incorrect description of pod_mutation_hook in kubernetes.rst

2020-05-08 Thread GitBox
ashb commented on a change in pull request #8757: URL: https://github.com/apache/airflow/pull/8757#discussion_r422377856 ## File path: docs/kubernetes.rst ## @@ -40,11 +40,44 @@ that has the ability to mutate pod objects before sending them to the Kubernetes for scheduling.

[GitHub] [airflow] jvstein opened a new issue #8792: KubernetesPodOperator does not return XCOM on pod failure

2020-05-08 Thread GitBox
jvstein opened a new issue #8792: URL: https://github.com/apache/airflow/issues/8792 **Apache Airflow version**: 1.10.9 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.14.9 **Environment**: - **Cloud provider or hardware

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8792: KubernetesPodOperator does not return XCOM on pod failure

2020-05-08 Thread GitBox
boring-cyborg[bot] commented on issue #8792: URL: https://github.com/apache/airflow/issues/8792#issuecomment-626055393 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] peter-volkov opened a new pull request #8791: [8650] Add Yandex.Cloud custom connection to 1.10 for backported provider

2020-05-08 Thread GitBox
peter-volkov opened a new pull request #8791: URL: https://github.com/apache/airflow/pull/8791 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [ ] Unit tests coverage for changes (not needed for

[GitHub] [airflow] dimon222 commented on issue #8212: Can't read S3 remote logs using Airflow 1.10.9

2020-05-08 Thread GitBox
dimon222 commented on issue #8212: URL: https://github.com/apache/airflow/issues/8212#issuecomment-625964491 @webster-chainalysis Worker class switch in airflow.cfg from gevent/eventlet to "sync" seem to resolve the issue for me.

[airflow] branch master updated: fix typing errors reported by dmypy (#8773)

2020-05-08 Thread ash
This is an automated email from the ASF dual-hosted git repository. ash pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new a091c1f fix typing errors reported by dmypy

[GitHub] [airflow] ashb commented on a change in pull request #8701: Adding ElastiCache Hook for creating, describing and deleting replication groups

2020-05-08 Thread GitBox
ashb commented on a change in pull request #8701: URL: https://github.com/apache/airflow/pull/8701#discussion_r422410165 ## File path: airflow/providers/amazon/aws/hooks/elasticache.py ## @@ -0,0 +1,238 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[GitHub] [airflow] ashb commented on a change in pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
ashb commented on a change in pull request #8772: URL: https://github.com/apache/airflow/pull/8772#discussion_r422375825 ## File path: airflow/serialization/serialized_objects.py ## @@ -395,6 +411,59 @@ def deserialize_operator(cls, encoded_op: Dict[str, Any]) ->

[GitHub] [airflow] j-y-matsubara commented on a change in pull request #8757: Fix the incorrect description of pod_mutation_hook in kubernetes.rst

2020-05-08 Thread GitBox
j-y-matsubara commented on a change in pull request #8757: URL: https://github.com/apache/airflow/pull/8757#discussion_r422455718 ## File path: docs/kubernetes.rst ## @@ -40,11 +40,44 @@ that has the ability to mutate pod objects before sending them to the Kubernetes for

  1   2   >