[GitHub] [airflow] dorranh commented on issue #7428: [AIRFLOW-4526] Poll k8s logs to avoid problem with kubernetes logs --follow

[GitBox]

dorranh commented on issue #7428: [AIRFLOW-4526] Poll k8s logs to avoid problem 
with kubernetes logs --follow
URL: https://github.com/apache/airflow/pull/7428#issuecomment-615069150
 
 
   Hi @sarah-johnson, thanks for putting together this PR. We're still having 
to hack around this issue with a custom KubernetesPodOperator on my team. Is 
the main holdup on this merged getting tests to pass? If so, I'm happy to try 
and help get them running.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ivorynoise commented on issue #7724: [AIRFLOW-1536] Inherit umask from parent process in daemon mode

[GitBox]

ivorynoise commented on issue #7724: [AIRFLOW-1536] Inherit umask from parent 
process in daemon mode
URL: https://github.com/apache/airflow/pull/7724#issuecomment-615068460
 
 
   Yeah sure. I will do it today. Was quite occupied with release at my
   company.
   
   On Wed, 15 Apr 2020 at 14:54, Jarek Potiuk  wrote:
   
   > @ivorynoise  -> my bad. I totally missed
   > that one. Can you please reabase and I will be happy to merge it!
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > , or
   > unsubscribe
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] irvifa opened a new issue #8416: Clustered Table is Not supported in BQ Operator Airflow 1.10.1

[GitBox]

irvifa opened a new issue #8416: Clustered Table is Not supported in BQ 
Operator Airflow 1.10.1
URL: https://github.com/apache/airflow/issues/8416
 
 
   I got the following error when try to dump the data to put the BQ result 
because of the fact that my table have a partition:
   
   ```
   Expects partitioning specification interval(type:day)[clustering index], but 
input partitioning specification is interval(type:day) cluster
   ```
   
   Even though I give my `cluster_fields` in BQ operator it's still not 
working. Is ithis expected, because it works in 1.10.3.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8416: Clustered Table is Not supported in BQ Operator Airflow 1.10.1

[GitBox]

boring-cyborg[bot] commented on issue #8416: Clustered Table is Not supported 
in BQ Operator Airflow 1.10.1
URL: https://github.com/apache/airflow/issues/8416#issuecomment-615054723
 
 
   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] haoliang7 opened a new pull request #8415: Fix subcommand error when running production image without argument

[GitBox]

haoliang7 opened a new pull request #8415: Fix subcommand error when running 
production image without argument
URL: https://github.com/apache/airflow/pull/8415
 
 
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj opened a new pull request #8414: Use repeated arguments in pytest

[GitBox]

mik-laj opened a new pull request #8414: Use repeated arguments in pytest
URL: https://github.com/apache/airflow/pull/8414
 
 
   The comma-separated list as an argument is very uncommon.  This caused me a 
problem, because the natural impulse suggests that the repeated arguments 
should add up values.
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [X] Description above provides context of the change
   - [X] Unit tests coverage for changes (not needed for documentation changes)
   - [X] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [X] Relevant documentation is updated including usage instructions.
   - [X] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil opened a new pull request #8413: Add 1.10.10 SecretsBackends contrib paths with Deprecation Warning

[GitBox]

kaxil opened a new pull request #8413: Add 1.10.10 SecretsBackends contrib 
paths with Deprecation Warning
URL: https://github.com/apache/airflow/pull/8413
 
 
   Add 1.10.10 SecretsBackends contrib paths with Deprecation Warning
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] TeddyHartanto commented on issue #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream

[GitBox]

TeddyHartanto commented on issue #7735: [AIRFLOW-4549] Allow skipped tasks to 
satisfy wait_for_downstream
URL: https://github.com/apache/airflow/pull/7735#issuecomment-614993827
 
 
   Hi @ashb, I've fixed the tests :) please let me know if there's anything 
else that I miss


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil opened a new pull request #8412: Remove duplicate dependency ('curl') from Dockerfile

[GitBox]

kaxil opened a new pull request #8412: Remove duplicate dependency ('curl') 
from Dockerfile
URL: https://github.com/apache/airflow/pull/8412
 
 
   Unless I am missing something we already install curl at 
https://github.com/apache/airflow/blob/5e3a831c8e4521d22f553d3c1b57a093c9de4199/Dockerfile#L71-L78
   
   so we don't need to install it again
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil opened a new pull request #8411: BugFix: DAG trigger via UI error in RBAC UI

[GitBox]

kaxil opened a new pull request #8411: BugFix: DAG trigger via UI error in RBAC 
UI
URL: https://github.com/apache/airflow/pull/8411
 
 
   resolves https://github.com/apache/airflow/issues/8247
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6852) Force rescheule mode for sensors when running on SequentialExecutor to avoid deadlocks.

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085351#comment-17085351
 ] 

ASF GitHub Bot commented on AIRFLOW-6852:
-

stale[bot] commented on pull request #7473: [AIRFLOW-6852] Force reschedule 
mode for sensors when running on SequentialExecutor
URL: https://github.com/apache/airflow/pull/7473
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Force rescheule mode for sensors when running on SequentialExecutor to avoid 
> deadlocks.
> ---
>
> Key: AIRFLOW-6852
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6852
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: executors
>Affects Versions: 2.0.0
>Reporter: Aleksander Nitecki
>Assignee: Aleksander Nitecki
>Priority: Major
>  Labels: pull-request-available
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6834) Fix flaky test test_scheduler_job.py::TestDagFileProcessor::test_dag_file_processor_process_task_instances_depends_on_past

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085352#comment-17085352
 ] 

ASF GitHub Bot commented on AIRFLOW-6834:
-

stale[bot] commented on pull request #7470: [AIRFLOW-6834] Fix some flaky tests 
in test_scheduler_job.py
URL: https://github.com/apache/airflow/pull/7470
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix flaky test 
> test_scheduler_job.py::TestDagFileProcessor::test_dag_file_processor_process_task_instances_depends_on_past
> --
>
> Key: AIRFLOW-6834
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6834
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: tests
>Affects Versions: 1.10.9
>Reporter: Qian Yu
>Assignee: Qian Yu
>Priority: Major
>
> test_scheduler_job.py has a few flaky tests. Some are marked with 
> pytest.mark.xfail, but this one is not marked flaky. It sometimes fails in 
> Travis.
>  
> Example 1:
>  
> {code:python}
>  FAILURES 
> 
> _ 
> TestDagFileProcessor.test_dag_file_processor_process_task_instances_depends_on_past_0
>  __
> a = ( testMethod=test_dag_file_processor_process_task_instances_depends_on_past_0>,)
> @wraps(func)
> def standalone_func(*a):
> >   return func(*(a + p.args), **p.kwargs)
> /usr/local/lib/python3.6/site-packages/parameterized/parameterized.py:518:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> self =  testMethod=test_dag_file_processor_process_task_instances_depends_on_past_0>
> state = None, start_date = None, end_date = None
> @parameterized.expand([
> [State.NONE, None, None],
> [State.UP_FOR_RETRY, timezone.utcnow() - 
> datetime.timedelta(minutes=30),
>  timezone.utcnow() - datetime.timedelta(minutes=15)],
> [State.UP_FOR_RESCHEDULE, timezone.utcnow() - 
> datetime.timedelta(minutes=30),
>  timezone.utcnow() - datetime.timedelta(minutes=15)],
> ])
> def test_dag_file_processor_process_task_instances_depends_on_past(self, 
> state, start_date, end_date):
> """
> Test if _process_task_instances puts the right task instances into the
> mock_list.
> """
> dag = DAG(
> dag_id='test_scheduler_process_execute_task_depends_on_past',
> start_date=DEFAULT_DATE,
> default_args={
> 'depends_on_past': True,
> },
> )
> dag_task1 = DummyOperator(
> task_id='dummy1',
> dag=dag,
> owner='airflow')
> dag_task2 = DummyOperator(
> task_id='dummy2',
> dag=dag,
> owner='airflow')
> with create_session() as session:
> orm_dag = DagModel(dag_id=dag.dag_id)
> session.merge(orm_dag)
> dag_file_processor = DagFileProcessor(dag_ids=[], 
> log=mock.MagicMock())
> dag.clear()
> dr = dag_file_processor.create_dag_run(dag)
> self.assertIsNotNone(dr)
> with create_session() as session:
> tis = dr.get_task_instances(session=session)
> for ti in tis:
> ti.state = state
> ti.start_date = start_date
> ti.end_date = end_date
> ti_to_schedule = []
> dag_file_processor._process_task_instances(dag, 
> task_instances_list=ti_to_schedule)
> >   assert ti_to_schedule == [
> (dag.dag_id, dag_task1.task_id, DEFAULT_DATE, TRY_NUMBER),
> (dag.dag_id, dag_task2.task_id, DEFAULT_DATE, TRY_NUMBER),
> ]
> E   AssertionError: assert 
> [('test_scheduler_process_execute_task_depends_on_past',\n  'dummy2',\n  
> datetime.datetime(2016, 1, 1, 0, 0, tzinfo= +00:00:00, STD]>),\n  1),\n 
> ('test_scheduler_process_execute_task_depends_on_past',\n  'dummy1',\n  
> datetime.datetime(2016, 1, 1, 0, 0, tzinfo= +00:00:00, STD]>),\n  1)] == 
> [('test_scheduler_process_execute_task_depends_on_past',\n  'dummy1',\n  
> datetime.datetime(2016, 1, 1, 0, 0, tzinfo=),\n  1),\n 
> ('test_scheduler_process_execute_task_depends_on_past',\n  'dummy2',\n  
> datetime.datetime(2016, 1, 1, 0, 0, tzinfo=),\n  1)]
> E At index 0 diff: 
> ('test_scheduler_proces

[jira] [Commented] (AIRFLOW-5840) Add a link to the external task that ExternalTaskSensor is waiting for

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085350#comment-17085350
 ] 

ASF GitHub Bot commented on AIRFLOW-5840:
-

stale[bot] commented on pull request #7119: [AIRFLOW-5840] Add operator extra 
link to external task sensor
URL: https://github.com/apache/airflow/pull/7119
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add a link to the external task that ExternalTaskSensor is waiting for
> --
>
> Key: AIRFLOW-5840
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5840
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Affects Versions: 1.10.6
>Reporter: Qian Yu
>Assignee: Robert Conroy
>Priority: Major
>  Labels: ExternalTaskSensor, Target, get_link, link
>
> When using ExternalTaskSensor, there's no indication on the Web UI regarding 
> which task the sensor is waiting for. Does anyone have ideas for improving 
> this? 
> One idea is to override the get_link() abstract method of ExternalTaskSensor 
> and point it to the task that the sensor waits for. Maybe something like this 
> (just for demo, not tested and likely not working):
> {code:python}
> def get_link(self, operator: BaseOperator, dttm: datetime) -> str:
>  return 
> "{}?dag_id={}&task_id={}&execution_date={}".format(conf["base_url"], 
> self.external_task_id, self.execution_date.isoformat())
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] stale[bot] closed pull request #7470: [AIRFLOW-6834] Fix some flaky tests in test_scheduler_job.py

[GitBox]

stale[bot] closed pull request #7470: [AIRFLOW-6834] Fix some flaky tests in 
test_scheduler_job.py
URL: https://github.com/apache/airflow/pull/7470
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] stale[bot] commented on issue #7428: [AIRFLOW-4526] Poll k8s logs to avoid problem with kubernetes logs --follow

[GitBox]

stale[bot] commented on issue #7428: [AIRFLOW-4526] Poll k8s logs to avoid 
problem with kubernetes logs --follow
URL: https://github.com/apache/airflow/pull/7428#issuecomment-614974503
 
 
   This issue has been automatically marked as stale because it has not had 
recent activity. It will be closed if no further activity occurs. Thank you for 
your contributions.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] stale[bot] closed pull request #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor

[GitBox]

stale[bot] closed pull request #7473: [AIRFLOW-6852] Force reschedule mode for 
sensors when running on SequentialExecutor
URL: https://github.com/apache/airflow/pull/7473
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] stale[bot] closed pull request #7119: [AIRFLOW-5840] Add operator extra link to external task sensor

[GitBox]

stale[bot] closed pull request #7119: [AIRFLOW-5840] Add operator extra link to 
external task sensor
URL: https://github.com/apache/airflow/pull/7119
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil edited a comment on issue #8247: Serialized DAG trigger via UI error - AttributeError: 'NoneType' object has no attribute 'create_dagrun'

[GitBox]

kaxil edited a comment on issue #8247: Serialized DAG trigger via UI error - 
AttributeError: 'NoneType' object has no attribute 'create_dagrun'
URL: https://github.com/apache/airflow/issues/8247#issuecomment-614970236
 
 
   Meanwhile, can you try the following docker image and let me know if that 
solves the issue for you:
   
   ```
   docker pull kaxil/docker-airflow:1.10.10
   ```
   
   Only use the non-rbac UI


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #8247: Serialized DAG trigger via UI error - AttributeError: 'NoneType' object has no attribute 'create_dagrun'

[GitBox]

kaxil commented on issue #8247: Serialized DAG trigger via UI error - 
AttributeError: 'NoneType' object has no attribute 'create_dagrun'
URL: https://github.com/apache/airflow/issues/8247#issuecomment-614970236
 
 
   Meanwhile, can you try the following docker image and let me know if that 
solves the issue for you:
   
   ```
   docker pull kaxil/docker-airflow:1.10.10
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #8247: Serialized DAG trigger via UI error - AttributeError: 'NoneType' object has no attribute 'create_dagrun'

[GitBox]

kaxil commented on issue #8247: Serialized DAG trigger via UI error - 
AttributeError: 'NoneType' object has no attribute 'create_dagrun'
URL: https://github.com/apache/airflow/issues/8247#issuecomment-614968274
 
 
   OK, I could reproduce this. I will create a PR to fix this but would be 
available in Airflow 1.10.11.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jstern commented on issue #7133: [AIRFLOW-6535] add exception to fail without retry

[GitBox]

jstern commented on issue #7133: [AIRFLOW-6535] add exception to fail without 
retry
URL: https://github.com/apache/airflow/pull/7133#issuecomment-614967805
 
 
   > gentle ping @jstern
   
   hi @tooptoop4 ! just waiting on https://github.com/apache/airflow/pull/8095 
to hit master so the docs can build cleanly and then hopefully we'll be able to 
get this one ready to merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5577) Dags Filter_by_owner is missing in RBAC

[Hari (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085324#comment-17085324
 ] 

Hari commented on AIRFLOW-5577:
---

I agree that in RBAC has dag level permission, you can think as an enterprise 
level. If you have many users using airflow, and each user are creating 100+ 
dags. How many admins required to maintain the cluster to provide dag level 
permission. Instead of that if we provide filter by user in Airflow or add the 
Filter by user in FAB, that would be great for easy maintenance.

> Dags Filter_by_owner is missing in RBAC
> ---
>
> Key: AIRFLOW-5577
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5577
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.3, 1.10.4, 1.10.5
>Reporter: Hari
>Assignee: Hari
>Priority: Major
>  Labels: easyfix
>
> After enabling the RBAC, the dags filter by owner option is missing. All the 
> Dags will be visible to all the users.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8396: Fix error message in production entrypoint.sh

[GitBox]

boring-cyborg[bot] commented on issue #8396: Fix error message in production 
entrypoint.sh
URL: https://github.com/apache/airflow/pull/8396#issuecomment-614962044
 
 
   Awesome work, congrats on your first merged pull request!
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil merged pull request #8396: Fix error message in production entrypoint.sh

[GitBox]

kaxil merged pull request #8396: Fix error message in production entrypoint.sh
URL: https://github.com/apache/airflow/pull/8396
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5577) Dags Filter_by_owner is missing in RBAC

[Kaxil Naik (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085318#comment-17085318
 ] 

Kaxil Naik commented on AIRFLOW-5577:
-

Check: https://airflow.readthedocs.io/en/latest/security.html#dag-level-role

You can give permissions to certain users read permissions on specific dags

> Dags Filter_by_owner is missing in RBAC
> ---
>
> Key: AIRFLOW-5577
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5577
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.3, 1.10.4, 1.10.5
>Reporter: Hari
>Assignee: Hari
>Priority: Major
>  Labels: easyfix
>
> After enabling the RBAC, the dags filter by owner option is missing. All the 
> Dags will be visible to all the users.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io edited a comment on issue #8410: Resolve Cross-site Scripting (XSS) vulnerability SNYK-JS-JQUERY-565129

[GitBox]

codecov-io edited a comment on issue #8410: Resolve Cross-site Scripting (XSS) 
vulnerability SNYK-JS-JQUERY-565129
URL: https://github.com/apache/airflow/pull/8410#issuecomment-614951666
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8410?src=pr&el=h1) 
Report
   > Merging 
[#8410](https://codecov.io/gh/apache/airflow/pull/8410?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/44ddf54adf7cfe57bfea98cd2726152a2ba19e18&el=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/8410/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8410?src=pr&el=tree)
   
   ```diff
   @@  Coverage Diff   @@
   ##   master   #8410   +/-   ##
   ==
 Coverage6.24%   6.24%   
   ==
 Files 941 941   
 Lines   45606   45606   
   ==
 Hits 28462846   
 Misses  42760   42760   
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/8410?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/8410?src=pr&el=footer). 
Last update 
[44ddf54...6849cc5](https://codecov.io/gh/apache/airflow/pull/8410?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5577) Dags Filter_by_owner is missing in RBAC

[cdmikechen (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085314#comment-17085314
 ] 

cdmikechen commented on AIRFLOW-5577:
-

[~kaxilnaik] Thank you for answering.

I found RBAC just controlled by Roles and there are several initial 
initialization roles in Airflow. If I create a user by User/Viewer Role, this 
user can read all DAGs and it can't filter DAG by himself.

It does not achieve the purpose of filter_by_owner, or can you explain how to 
use RBAC to control different users to only see their DAG?

 

> Dags Filter_by_owner is missing in RBAC
> ---
>
> Key: AIRFLOW-5577
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5577
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 1.10.3, 1.10.4, 1.10.5
>Reporter: Hari
>Assignee: Hari
>Priority: Major
>  Labels: easyfix
>
> After enabling the RBAC, the dags filter by owner option is missing. All the 
> Dags will be visible to all the users.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io commented on issue #8410: Resolve Cross-site Scripting (XSS) vulnerability SNYK-JS-JQUERY-565129

[GitBox]

codecov-io commented on issue #8410: Resolve Cross-site Scripting (XSS) 
vulnerability SNYK-JS-JQUERY-565129
URL: https://github.com/apache/airflow/pull/8410#issuecomment-614951666
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8410?src=pr&el=h1) 
Report
   > Merging 
[#8410](https://codecov.io/gh/apache/airflow/pull/8410?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/44ddf54adf7cfe57bfea98cd2726152a2ba19e18&el=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/8410/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8410?src=pr&el=tree)
   
   ```diff
   @@  Coverage Diff   @@
   ##   master   #8410   +/-   ##
   ==
 Coverage6.24%   6.24%   
   ==
 Files 941 941   
 Lines   45606   45606   
   ==
 Hits 28462846   
 Misses  42760   42760   
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/8410?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/8410?src=pr&el=footer). 
Last update 
[44ddf54...6849cc5](https://codecov.io/gh/apache/airflow/pull/8410?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mbarrien commented on issue #8396: Fix error message in production entrypoint.sh

[GitBox]

mbarrien commented on issue #8396: Fix error message in production entrypoint.sh
URL: https://github.com/apache/airflow/pull/8396#issuecomment-614943944
 
 
   Repointed PR at master base branch, rebased and force-pushed. I will go 
ahead and check off the boxes above.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ryw opened a new pull request #8410: Resolve Cross-site Scripting (XSS) vulnerability SNYK-JS-JQUERY-565129

[GitBox]

ryw opened a new pull request #8410: Resolve Cross-site Scripting (XSS) 
vulnerability SNYK-JS-JQUERY-565129
URL: https://github.com/apache/airflow/pull/8410
 
 
   Resolves XSS CVE present in jquery < 3.5.0 
   https://app.snyk.io/vuln/SNYK-JS-JQUERY-565129 
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [X] Description above provides context of the change
   - [X] Unit tests coverage for changes (not needed for documentation changes)
   - [X] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [X] Relevant documentation is updated including usage instructions.
   - [X] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #8242: Jinjafy params arg in BaseOperator

[GitBox]

codecov-io commented on issue #8242: Jinjafy params arg in BaseOperator
URL: https://github.com/apache/airflow/pull/8242#issuecomment-614933246
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8242?src=pr&el=h1) 
Report
   > Merging 
[#8242](https://codecov.io/gh/apache/airflow/pull/8242?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/12b9b64e10626bb1bc382815edeac9cbedd2701a&el=desc)
 will **decrease** coverage by `54.90%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/8242/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8242?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff @@
   ##   master#8242   +/-   ##
   ===
   - Coverage   88.32%   33.42%   -54.91% 
   ===
 Files 936  941+5 
 Lines   4531945608  +289 
   ===
   - Hits4002915244-24785 
   - Misses   529030364+25074 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/8242?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/8242/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `56.85% <0.00%> (-38.84%)` | :arrow_down: |
   | 
[airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/8242/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5)
 | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | 
[airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/8242/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==)
 | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | 
[airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/8242/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=)
 | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | 
[airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/8242/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=)
 | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | 
[airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/8242/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=)
 | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | 
[airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/airflow/pull/8242/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5)
 | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | 
[airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/8242/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5)
 | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | 
[airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/8242/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5)
 | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | 
[airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/airflow/pull/8242/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5)
 | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [817 
more](https://codecov.io/gh/apache/airflow/pull/8242/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/8242?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/8242?src=pr&el=footer). 
Last update 
[12b9b64...234a3af](https://codecov.io/gh/apache/airflow/pull/8242?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7626: [AIRFLOW-5575] Cannot use the mark future success/failure feature in the UI

[GitBox]

codecov-io edited a comment on issue #7626: [AIRFLOW-5575] Cannot use the mark 
future success/failure feature in the UI
URL: https://github.com/apache/airflow/pull/7626#issuecomment-611226408
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=h1) 
Report
   > Merging 
[#7626](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/eca7fffbae48bfaae95208f8052ff177a60aac27&el=desc)
 will **decrease** coverage by `0.00%`.
   > The diff coverage is `20.00%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7626/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #7626  +/-   ##
   =
   - Coverage6.24%   6.24%   -0.01% 
   =
 Files 941 941  
 Lines   45606   45607   +1 
   =
 Hits 28462846  
   - Misses  42760   42761   +1 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5)
 | `26.65% <0.00%> (ø)` | |
   | 
[airflow/operators/latest\_only\_operator.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbGF0ZXN0X29ubHlfb3BlcmF0b3IucHk=)
 | `0.00% <0.00%> (ø)` | |
   | 
[...rflow/providers/google/cloud/operators/dataproc.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9kYXRhcHJvYy5weQ==)
 | `0.00% <ø> (ø)` | |
   | 
[airflow/serialization/serialized\_objects.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZXJpYWxpemF0aW9uL3NlcmlhbGl6ZWRfb2JqZWN0cy5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[airflow/utils/timezone.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy90aW1lem9uZS5weQ==)
 | `52.08% <0.00%> (ø)` | |
   | 
[airflow/www/decorators.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvZGVjb3JhdG9ycy5weQ==)
 | `0.00% <ø> (ø)` | |
   | 
[airflow/models/xcom.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMveGNvbS5weQ==)
 | `43.95% <100.00%> (ø)` | |
   | 
[airflow/settings.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==)
 | `73.61% <100.00%> (ø)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=footer). 
Last update 
[eca7fff...32ea31d](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7626: [AIRFLOW-5575] Cannot use the mark future success/failure feature in the UI

[GitBox]

codecov-io edited a comment on issue #7626: [AIRFLOW-5575] Cannot use the mark 
future success/failure feature in the UI
URL: https://github.com/apache/airflow/pull/7626#issuecomment-611226408
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=h1) 
Report
   > Merging 
[#7626](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/eca7fffbae48bfaae95208f8052ff177a60aac27&el=desc)
 will **decrease** coverage by `0.00%`.
   > The diff coverage is `20.00%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7626/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #7626  +/-   ##
   =
   - Coverage6.24%   6.24%   -0.01% 
   =
 Files 941 941  
 Lines   45606   45607   +1 
   =
 Hits 28462846  
   - Misses  42760   42761   +1 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5)
 | `26.65% <0.00%> (ø)` | |
   | 
[airflow/operators/latest\_only\_operator.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbGF0ZXN0X29ubHlfb3BlcmF0b3IucHk=)
 | `0.00% <0.00%> (ø)` | |
   | 
[...rflow/providers/google/cloud/operators/dataproc.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9kYXRhcHJvYy5weQ==)
 | `0.00% <ø> (ø)` | |
   | 
[airflow/serialization/serialized\_objects.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZXJpYWxpemF0aW9uL3NlcmlhbGl6ZWRfb2JqZWN0cy5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[airflow/utils/timezone.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy90aW1lem9uZS5weQ==)
 | `52.08% <0.00%> (ø)` | |
   | 
[airflow/www/decorators.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvZGVjb3JhdG9ycy5weQ==)
 | `0.00% <ø> (ø)` | |
   | 
[airflow/models/xcom.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMveGNvbS5weQ==)
 | `43.95% <100.00%> (ø)` | |
   | 
[airflow/settings.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==)
 | `73.61% <100.00%> (ø)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=footer). 
Last update 
[eca7fff...32ea31d](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #8305: Guide for Apache Spark operators

[GitBox]

kaxil commented on issue #8305: Guide for Apache Spark operators
URL: https://github.com/apache/airflow/pull/8305#issuecomment-614927255
 
 
   Static tests are failing, you might want to apply following diff:
   
   ```diff
   
   diff --git a/docs/howto/connection/spark.rst 
b/docs/howto/connection/spark.rst
   index bc68bfc3f..d52098cb3 100644
   --- a/docs/howto/connection/spark.rst
   +++ b/docs/howto/connection/spark.rst
   @@ -38,6 +38,6 @@ Extra (optional)

* ``queue`` - The name of the YARN queue to which the application is 
submitted.
* ``deploy-mode`` - Whether to deploy your driver on the worker nodes 
(cluster) or locally as an external client (client).
   -* ``spark-home`` - If passed then build the ``spark-binary`` executable 
path using it (``spark-home``/bin/``spark-binary``); otherwise assume that 
``spark-binary`` is present in the PATH of the executing user. 
   +* ``spark-home`` - If passed then build the ``spark-binary`` executable 
path using it (``spark-home``/bin/``spark-binary``); otherwise assume that 
``spark-binary`` is present in the PATH of the executing user.
* ``spark-binary`` - The command to use for Spark submit. Some distros 
may use ``spark2-submit``. Default ``spark-submit``.
* ``namespace`` - Kubernetes namespace (``spark.kubernetes.namespace``) 
to divide cluster resources between multiple users (via resource quota).
   diff --git a/docs/howto/operator/apache/spark.rst 
b/docs/howto/operator/apache/spark.rst
   index be52ddde6..b256d9d09 100644
   --- a/docs/howto/operator/apache/spark.rst
   +++ b/docs/howto/operator/apache/spark.rst
   @@ -38,4 +38,3 @@ SparkSqlOperator

SparkSubmitOperator
---
   -
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] brokenjacobs commented on a change in pull request #8287: [AIRFLOW-4363] Fix JSON encoding error

[GitBox]

brokenjacobs commented on a change in pull request #8287: [AIRFLOW-4363] Fix 
JSON encoding error
URL: https://github.com/apache/airflow/pull/8287#discussion_r409880823
 
 

 ##
 File path: airflow/providers/docker/operators/docker.py
 ##
 @@ -280,9 +279,10 @@ def execute(self, context):
 # Pull the docker image if `force_pull` is set or image does not exist 
locally
 if self.force_pull or not self.cli.images(name=self.image):
 self.log.info('Pulling docker image %s', self.image)
-for line in self.cli.pull(self.image, stream=True, decode=True):
-output = json.loads(line.decode('utf-8').strip())
-if 'status' in output:
+for output in self.cli.pull(self.image, stream=True, decode=True):
+if isinstance(output, str):
+self.log.info("%s", output)
+if isinstance(output, dict) and 'status' in output:
 
 Review comment:
   Looks much better. Is there ever a case where cli.pull will return a str 
when called with decode=True?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #8268: Update the tree view of dag on Concepts Last Run Only

[GitBox]

kaxil commented on issue #8268: Update the tree view of dag on Concepts Last 
Run Only
URL: https://github.com/apache/airflow/pull/8268#issuecomment-614924145
 
 
   Thanks, @boittega for your contribution :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8268: Update the tree view of dag on Concepts Last Run Only

[GitBox]

boring-cyborg[bot] commented on issue #8268: Update the tree view of dag on 
Concepts Last Run Only
URL: https://github.com/apache/airflow/pull/8268#issuecomment-614924034
 
 
   Awesome work, congrats on your first merged pull request!
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil merged pull request #8268: Update the tree view of dag on Concepts Last Run Only

[GitBox]

kaxil merged pull request #8268: Update the tree view of dag on Concepts Last 
Run Only
URL: https://github.com/apache/airflow/pull/8268
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil closed issue #8246: Docs: LatestOnlyOperator documentation is misleading

[GitBox]

kaxil closed issue #8246: Docs: LatestOnlyOperator documentation is misleading
URL: https://github.com/apache/airflow/issues/8246
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil merged pull request #8385: Catch ProgrammingError raised by MSSQL which doesn't know JSON_VALID()

[GitBox]

kaxil merged pull request #8385: Catch ProgrammingError raised by MSSQL which 
doesn't know JSON_VALID()
URL: https://github.com/apache/airflow/pull/8385
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] tooptoop4 commented on issue #7133: [AIRFLOW-6535] add exception to fail without retry

[GitBox]

tooptoop4 commented on issue #7133: [AIRFLOW-6535] add exception to fail 
without retry
URL: https://github.com/apache/airflow/pull/7133#issuecomment-614879429
 
 
   gentle ping @jstern 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #8396: Fix error message in production entrypoint.sh

[GitBox]

kaxil commented on issue #8396: Fix error message in production entrypoint.sh
URL: https://github.com/apache/airflow/pull/8396#issuecomment-614876191
 
 
   Yeah in this case, for Dockerfile, we don't need to any test or docs for 
this PR. But can you create a new PR to fix this issue in Master too please?
   
   
   The workflow is PR is against a Master and one of the Airflow Committers 
would then backport it to v1-10-* branch


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mbarrien commented on issue #8396: Fix error message in production entrypoint.sh

[GitBox]

mbarrien commented on issue #8396: Fix error message in production entrypoint.sh
URL: https://github.com/apache/airflow/pull/8396#issuecomment-614860696
 
 
   Master also does suffer from this, but I noticed this in my attempts to do 
1.10. Don't know what the convention is in this repo for PRs against master and 
backports.
   
   For the 2 unchecked checkboxes, I don't know if there are any unit tests 
available to test something like this (and it seems like quite the task to 
create one to test a Dockerfile). Also don't know what docs need updating for 
this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] alexandrecaze commented on issue #8282: Missing crossreferences between hooks and guides

[GitBox]

alexandrecaze commented on issue #8282: Missing crossreferences between hooks 
and guides
URL: https://github.com/apache/airflow/issues/8282#issuecomment-614839238
 
 
   Hi @mik-laj I am down to work on this issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj opened a new pull request #8409: Use less fancy tables in CLI by default

[GitBox]

mik-laj opened a new pull request #8409: Use less fancy tables in CLI by default
URL: https://github.com/apache/airflow/pull/8409
 
 
   Fancy grid is fancier but has the following disadvantages:
   * Takes up a lot more space, which makes reading difficult
  `AIRFLOW__CORE__DAGS_FOLDER=$(readlink -e 
airflow/providers/google/cloud/example_dags/) airflow dags list --output 
fancy_grid  | wc -l`=> 152
  `AIRFLOW__CORE__DAGS_FOLDER=$(readlink -e 
airflow/providers/google/cloud/example_dags/) airflow dags list --output plain  
| wc -l`=> 76
* Does not work with AWK and other Linux tools ( e.g. `airflow dags list  | 
awk '{print $2}'  | sort | uniq`). 
* Is less popular  - `gcloud` uses plain table
   
   On the other hand, the only advantages of a fancy grid being fancier.
   
   Before:
   ![Screenshot 2020-04-16 at 20 26 
02](https://user-images.githubusercontent.com/12058428/79492700-a2fcb280-8020-11ea-8fe0-e1c9b910b8c4.png)
   
   After:
   ![Screenshot 2020-04-16 at 20 24 
41](https://user-images.githubusercontent.com/12058428/79492732-adb74780-8020-11ea-9a42-bf52e45e1e10.png)
   
   Because fancy_grid was not able to be parsed by Linux tools, this is not a 
breaking change. 
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [X] Description above provides context of the change
   - [X] Unit tests coverage for changes (not needed for documentation changes)
   - [X] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [X] Relevant documentation is updated including usage instructions.
   - [X] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on issue #8400: Less frequent dockerhub builds

[GitBox]

dimberman commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614814456
 
 
   Hmmm so I think long term the INFRA ticket is our best hope. Would be great 
to be able to push from github actions. Really hoping apache lets us do that. 
   
   We could also do what this PR is proposing and just make it less time (like 
maybe if the build is 2-3 hours old?)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7626: [AIRFLOW-5575] Cannot use the mark future success/failure feature in the UI

[GitBox]

codecov-io edited a comment on issue #7626: [AIRFLOW-5575] Cannot use the mark 
future success/failure feature in the UI
URL: https://github.com/apache/airflow/pull/7626#issuecomment-611226408
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=h1) 
Report
   > Merging 
[#7626](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/eca7fffbae48bfaae95208f8052ff177a60aac27&el=desc)
 will **decrease** coverage by `0.00%`.
   > The diff coverage is `25.00%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7626/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #7626  +/-   ##
   =
   - Coverage6.24%   6.24%   -0.01% 
   =
 Files 941 941  
 Lines   45606   45607   +1 
   =
 Hits 28462846  
   - Misses  42760   42761   +1 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5)
 | `26.65% <0.00%> (ø)` | |
   | 
[airflow/operators/latest\_only\_operator.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbGF0ZXN0X29ubHlfb3BlcmF0b3IucHk=)
 | `0.00% <0.00%> (ø)` | |
   | 
[...rflow/providers/google/cloud/operators/dataproc.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9kYXRhcHJvYy5weQ==)
 | `0.00% <ø> (ø)` | |
   | 
[airflow/serialization/serialized\_objects.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZXJpYWxpemF0aW9uL3NlcmlhbGl6ZWRfb2JqZWN0cy5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[airflow/utils/timezone.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy90aW1lem9uZS5weQ==)
 | `52.08% <0.00%> (ø)` | |
   | 
[airflow/www/decorators.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvZGVjb3JhdG9ycy5weQ==)
 | `0.00% <ø> (ø)` | |
   | 
[airflow/models/xcom.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMveGNvbS5weQ==)
 | `43.95% <100.00%> (ø)` | |
   | 
[airflow/settings.py](https://codecov.io/gh/apache/airflow/pull/7626/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==)
 | `73.61% <100.00%> (ø)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=footer). 
Last update 
[eca7fff...1abd802](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

[GitBox]

potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614797872
 
 
   And if you ask - no - we cannot define our own secrets in Github Actions to 
store such URL. We have no permissions to do it. :( ... This is what is the 
INFRA ticket about as well...


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #8400: Less frequent dockerhub builds

[GitBox]

codecov-io edited a comment on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614628834
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=h1) 
Report
   > Merging 
[#8400](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/6d5710785204120b57dc61e351e12ae0b9f142b0&el=desc)
 will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/8400/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #8400  +/-   ##
   =
   - Coverage6.24%   6.24%   -0.01% 
   =
 Files 941 941  
 Lines   45573   45606  +33 
   =
 Hits 28462846  
   - Misses  42727   42760  +33 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/cli.py](https://codecov.io/gh/apache/airflow/pull/8400/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9jbGkucHk=)
 | `0.00% <0.00%> (ø)` | |
   | 
[airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8400/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[airflow/cli/commands/config\_command.py](https://codecov.io/gh/apache/airflow/pull/8400/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY29uZmlnX2NvbW1hbmQucHk=)
 | `0.00% <0.00%> (ø)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=footer). 
Last update 
[6d57107...2215ff8](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #8400: Less frequent dockerhub builds

[GitBox]

codecov-io edited a comment on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614628834
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=h1) 
Report
   > Merging 
[#8400](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/6d5710785204120b57dc61e351e12ae0b9f142b0&el=desc)
 will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/8400/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #8400  +/-   ##
   =
   - Coverage6.24%   6.24%   -0.01% 
   =
 Files 941 941  
 Lines   45573   45606  +33 
   =
 Hits 28462846  
   - Misses  42727   42760  +33 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/cli.py](https://codecov.io/gh/apache/airflow/pull/8400/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9jbGkucHk=)
 | `0.00% <0.00%> (ø)` | |
   | 
[airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8400/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==)
 | `0.00% <0.00%> (ø)` | |
   | 
[airflow/cli/commands/config\_command.py](https://codecov.io/gh/apache/airflow/pull/8400/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY29uZmlnX2NvbW1hbmQucHk=)
 | `0.00% <0.00%> (ø)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=footer). 
Last update 
[6d57107...2215ff8](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

[GitBox]

potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614796124
 
 
   I mean - > we CAN have a corn Job setup (wherever) calling that trigger URL 
every 2-3 hours But is it safe to store such URL somewhere? Is it something 
that is sustainable long term (who will take care that it is actually still 
working  :))  Who will watch the watcher. ? 
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

[GitBox]

potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614794174
 
 
   See yourself @dimberman -> maybe you can find how to do it. I believe you 
have access to https://hub.docker.com/repository/docker/apache/airflow :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

[GitBox]

potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614792771
 
 
   There is no such feature in Dockerhub :shrug: You can trigger it on 
   
   1) push to repo (Branch or Tag)
   2) externally called URL trigger - this can be scheduled but the URL is 
essentially a credential and can be called many times. We would need to have 
some external service calling the URL 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on issue #8400: Less frequent dockerhub builds

[GitBox]

dimberman commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614791092
 
 
   @potiuk perhaps we can do hourly/bi-hourly jobs? So instead of 
build-on-merge we just build at the top of the hour? (we could manually trigger 
if it's an important fix)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

[GitBox]

potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614787569
 
 
   This is our Dockerhub build confifguration ^^
   
   * 4 images to build from every master build
   * 8 images for v1-10-test
   
   Unfortuantely we have 1 worker queue from DockerHub that builds them :(
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

[GitBox]

potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614786735
 
 
   ![Screenshot from 2020-04-16 
19-23-12](https://user-images.githubusercontent.com/595491/79486944-c838f300-8017-11ea-8fbb-132bd45ef54e.png)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #8408: Fix building image manifest (was removed by accident with prod img)

[GitBox]

potiuk merged pull request #8408: Fix building image manifest (was removed by 
accident with prod img)
URL: https://github.com/apache/airflow/pull/8408
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #8400: Less frequent dockerhub builds

[GitBox]

potiuk edited a comment on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614783967
 
 
   @dimberman It always lags quite a bit as DockerHub is very slow just "check 
if the image needs to be rebuilt aka Docker build from cache" takes anything 
between 10 and 20 minutes.  Currently we build so many images (including the 
PROD one)  that when we have a few merges, the build queue grows faster than it 
is cleared and pretty much every day we end up in a situation that the build 
waits a long time (700mins = 12 hours is pretty normal if we have a series of 
master merges) + if we have a v1-10-stable push, that's even longer. 
   
   From CI point of view it's OK. The whole system is built in the way that if 
the image is old, it - at most - run CI a bit slower. Assuming that 
master/v1-10-stable are constantly moving, I do not think this is a big issue 
to add such "10 hours" skip. 
   
   I also opened INFRA ticket to see if we can push the image from Github 
Actions (see my comment below. But I doubt it will be OK from the security POV:
   > Currently we have quite a lot of builds happening in DockerHub (including 
production image). It takes a long time to build the images and they are 
queuing now. I have created an infra ticket to ask if we can push images from 
Github Actions 
(https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20124) but I have 
no high hopes for it - it requires storing secrets in GA. That's why I propose 
that build scripts will check if the image is <10 hours old and only actually 
build it if it is.
   
   Alternatively we might want to rebuild the images regularly - every night 
for example, but this is a bit problematic (I explained it to @turbaszek above) 
- we could generate a trigger URL and use it in an external service (there is 
no cron build support in DockerHub. But I am not sure it's safe.
   
   I realise it is not perfect, do you have any ideas how we can do it 
differently ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

[GitBox]

potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614783967
 
 
   @dimberman It always lags quite a bit as DockerHub is very slow just "check 
if the image needs to reb uilt" takes anything between 10 and 20 minutes.  
Currently we build so many images (including the PROD one)  that when we have a 
few merges, the build queue grows faster than it is cleared and pretty much 
every day we end up in a situation that the build waits a long time (700mins = 
12 hours is pretty normal if we have a series of master merges) + if we have a 
v1-10-stable push, that's even longer. 
   
   From CI point of view it's OK. The whole system is built in the way that if 
the image is old, it - at most - run CI a bit slower. Assuming that 
master/v1-10-stable are constantly moving, I do not think this is a big issue 
to add such "10 hours" skip. 
   
   I also opened INFRA ticket to see if we can push the image from Github 
Actions (see my comment below. But I doubt it will be OK from the security POV:
   > Currently we have quite a lot of builds happening in DockerHub (including 
production image). It takes a long time to build the images and they are 
queuing now. I have created an infra ticket to ask if we can push images from 
Github Actions 
(https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20124) but I have 
no high hopes for it - it requires storing secrets in GA. That's why I propose 
that build scripts will check if the image is <10 hours old and only actually 
build it if it is.
   
   Alternatively we might want to rebuild the images regularly - every night 
for example, but this is a bit problematic (I explained it to @turbaszek above) 
- we could generate a trigger URL and use it in an external service (there is 
no cron build support in DockerHub. But I am not sure it's safe.
   
   I realise it is not perfect, do you have any ideas how we can do it 
differently ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #8313: Simplify handling of downloaded files in Dataflow

[GitBox]

mik-laj commented on issue #8313: Simplify handling of downloaded files in 
Dataflow
URL: https://github.com/apache/airflow/pull/8313#issuecomment-614780207
 
 
   @turbaszek ping 🐈  Can I ask for review again?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #8396: Fix error message in production entrypoint.sh

[GitBox]

kaxil commented on issue #8396: Fix error message in production entrypoint.sh
URL: https://github.com/apache/airflow/pull/8396#issuecomment-614778797
 
 
   This PR is against v1-10-stable, does Master not suffer from the same issue?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8408: Fix building image manifest (was removed by accident with prod img)

[GitBox]

potiuk commented on issue #8408: Fix building image manifest (was removed by 
accident with prod img)
URL: https://github.com/apache/airflow/pull/8408#issuecomment-614776508
 
 
   I must have removed it by accident when adding Prod image. Now building CI 
image with 'build-image' command will also build manifest that is used to 
compare if there is a new image to download


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk opened a new pull request #8408: Fix building image manifest (was removed by accident with prod img)

[GitBox]

potiuk opened a new pull request #8408: Fix building image manifest (was 
removed by accident with prod img)
URL: https://github.com/apache/airflow/pull/8408
 
 
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #8270: Provide official airflow chart

[GitBox]

ashb commented on issue #8270: Provide official airflow chart 
URL: https://github.com/apache/airflow/issues/8270#issuecomment-614772584
 
 
   We are actively working on making 
https://github.com/astronomer/airflow-chart/ suitable for wider use


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on issue #8400: Less frequent dockerhub builds

[GitBox]

dimberman commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614771005
 
 
   @potiuk Should we only be building master every 10 hours? I feel like master 
should be up-to-date with whatever the master branch is doing.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on a change in pull request #8393: Bring back CI optimisations

[GitBox]

dimberman commented on a change in pull request #8393: Bring back CI 
optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409694983
 
 

 ##
 File path: .github/workflows/ci.yml
 ##
 @@ -80,134 +80,152 @@ jobs:
 name: Build docs
 runs-on: ubuntu-latest
 env:
-  TRAVIS_JOB_NAME: "Build documentation"
   PYTHON_VERSION: 3.6
+  CI_JOB_TYPE: "Documentation"
 steps:
   - uses: actions/checkout@master
-  - name: "Build documentation"
+  - name: "Build CI image"
+run: ./scripts/ci/ci_prepare_image_on_ci.sh
+  - name: "Build docs"
 run: ./scripts/ci/ci_docs.sh
 
   tests-p36-postgres-integrations:
-name: "Tests [Postgres9.6][Py3.6][integrations]"
+name: "[Pg9.6][Py3.6][integrations]"
 runs-on: ubuntu-latest
 needs: [statics, statics-tests]
 env:
-  TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][integrations]"
   BACKEND: postgres
   PYTHON_VERSION: 3.6
   POSTGRES_VERSION: 9.6
   ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
   RUN_INTEGRATION_TESTS: all
+  CI_JOB_TYPE: "Tests"
 steps:
   - uses: actions/checkout@master
-  - name: "Tests [Postgres9.6][Py3.6][integrations]"
+  - name: "Build CI image"
+run: ./scripts/ci/ci_prepare_image_on_ci.sh
+  - name: "Tests"
 run: ./scripts/ci/ci_run_airflow_testing.sh
 
   tests-p36-postgres-providers:
 
 Review comment:
   We can do all of this using a matrix
   
   ```
   strategy:
 matrix:
   postgres: [Pg9, Pg10]
   python: [3.6, 3.7]
   strategy: [providers, core]
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8400: Less frequent dockerhub builds

[GitBox]

kaxil commented on a change in pull request #8400: Less frequent dockerhub 
builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409669217
 
 

 ##
 File path: scripts/ci/ci_build_dockerhub.sh
 ##
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ 
.CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ 
.*weeks.* ||
+  ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ 
.*years.* ||
+  ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   ```bash
   ❯ export tta="9 hours"
   ❯ if [[ "$tta" =~ [0-9]?\ hours ]]; then
   echo "Rebuilding the image."
   else
   echo "Skipping rebuilding the image."
   fi
   Rebuilding the image.
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman merged pull request #8407: Fixes check_migrations commands

[GitBox]

dimberman merged pull request #8407: Fixes check_migrations commands
URL: https://github.com/apache/airflow/pull/8407
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb merged pull request #8398: Allow "falsey" default arguments in CLI Parser

[GitBox]

ashb merged pull request #8398: Allow "falsey" default arguments in CLI Parser
URL: https://github.com/apache/airflow/pull/8398
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #8398: Allow "falsey" default arguments in CLI Parser

[GitBox]

ashb commented on issue #8398: Allow "falsey" default arguments in CLI Parser
URL: https://github.com/apache/airflow/pull/8398#issuecomment-614750849
 
 
   Bored waiting for travis to run the K8s tests for a cli change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

[GitBox]

potiuk commented on a change in pull request #8400: Less frequent dockerhub 
builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409677204
 
 

 ##
 File path: scripts/ci/ci_build_dockerhub.sh
 ##
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ 
.CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ 
.*weeks.* ||
+  ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ 
.*years.* ||
+  ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   I am not sure how the 10 hours are calculated - i.e. I think 9h + 1 minute 
might be displayed as 10 hours 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

[GitBox]

potiuk commented on a change in pull request #8400: Less frequent dockerhub 
builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409677558
 
 

 ##
 File path: scripts/ci/ci_build_dockerhub.sh
 ##
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ 
.CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ 
.*weeks.* ||
+  ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ 
.*years.* ||
+  ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   I wanted to make aproper calculation from UTC time but doing it in bash is 
kinda complex


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

[GitBox]

potiuk commented on a change in pull request #8400: Less frequent dockerhub 
builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409676501
 
 

 ##
 File path: scripts/ci/ci_build_dockerhub.sh
 ##
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ 
.CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ 
.*weeks.* ||
+  ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ 
.*years.* ||
+  ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   So basically I want to rebuild for everything = or above 10 hours.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

[GitBox]

potiuk commented on a change in pull request #8400: Less frequent dockerhub 
builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409676031
 
 

 ##
 File path: scripts/ci/ci_build_dockerhub.sh
 ##
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ 
.CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ 
.*weeks.* ||
+  ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ 
.*years.* ||
+  ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   9 hours is correct -> skipping


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

[GitBox]

potiuk commented on a change in pull request #8400: Less frequent dockerhub 
builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409675720
 
 

 ##
 File path: scripts/ci/ci_build_dockerhub.sh
 ##
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ 
.CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ 
.*weeks.* ||
+  ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ 
.*years.* ||
+  ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   But that's OK. I wanted to rebuild it when we hit 10 ..


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on issue #8407: Fixes check_migrations commands

[GitBox]

dimberman commented on issue #8407: Fixes check_migrations commands
URL: https://github.com/apache/airflow/pull/8407#issuecomment-614740692
 
 
   @kaxil we don't test it because it's so high up. (we test the the cli 
commands work, but this is the function that lazy loads the cli command. We'd 
need to do integration tests to test this (which I'm not against but isn't my 
top priority right now)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8400: Less frequent dockerhub builds

[GitBox]

kaxil commented on a change in pull request #8400: Less frequent dockerhub 
builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409668928
 
 

 ##
 File path: scripts/ci/ci_build_dockerhub.sh
 ##
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ 
.CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ 
.*weeks.* ||
+  ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ 
.*years.* ||
+  ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   Maybe we need the following:
   
   ```suggestion
 ${IMAGE_CREATED_SINCE} =~ [0-9]?\ hours ]]; then
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8400: Less frequent dockerhub builds

[GitBox]

kaxil commented on a change in pull request #8400: Less frequent dockerhub 
builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409669217
 
 

 ##
 File path: scripts/ci/ci_build_dockerhub.sh
 ##
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ 
.CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ 
.*weeks.* ||
+  ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ 
.*years.* ||
+  ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   ```bash
   ❯ export tta="9 hours"
   ❯ if [[ "$tta" =~ [0-9]?\ hours ]]; then
   echo "Rebuilding the image."
   else
   echo "Skipping rebuilding the image."
   fi
   Rebuilding the image.
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8400: Less frequent dockerhub builds

[GitBox]

kaxil commented on a change in pull request #8400: Less frequent dockerhub 
builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409667116
 
 

 ##
 File path: scripts/ci/ci_build_dockerhub.sh
 ##
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ 
.CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ 
.*weeks.* ||
+  ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ 
.*years.* ||
+  ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   ```bash
   ❯ export tta="9 hours"
   ❯ if [[ "$tta" =~ [0-9][0-9]\ hours ]]; then
   then> echo "Rebuilding the image."
   then> else
   else> echo "Skipping rebuilding the image."
   else> fi
   Skipping rebuilding the image.
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #8403: Test script tries to reset db regardless of db check

[GitBox]

potiuk merged pull request #8403: Test script tries to reset db regardless of 
db check
URL: https://github.com/apache/airflow/pull/8403
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

[GitBox]

potiuk commented on a change in pull request #8400: Less frequent dockerhub 
builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409659094
 
 

 ##
 File path: scripts/ci/ci_build_dockerhub.sh
 ##
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ 
.CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ 
.*weeks.* ||
+  ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ 
.*years.* ||
+  ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   Nope. It's 9 hours :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

[GitBox]

potiuk commented on a change in pull request #8400: Less frequent dockerhub 
builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409659454
 
 

 ##
 File path: scripts/ci/ci_build_dockerhub.sh
 ##
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ 
.CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ 
.*weeks.* ||
+  ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ 
.*years.* ||
+  ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   Then 48 hours and above that there are days.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk closed issue #8406: Unable to install apache-airflow 1.10.5

[GitBox]

potiuk closed issue #8406: Unable to install apache-airflow 1.10.5 
URL: https://github.com/apache/airflow/issues/8406
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8406: Unable to install apache-airflow 1.10.5

[GitBox]

potiuk commented on issue #8406: Unable to install apache-airflow 1.10.5 
URL: https://github.com/apache/airflow/issues/8406#issuecomment-614731512
 
 
   Please take a look at the info explained when you open the issue. For 
general troubleshooting and questions you should use Airflow  slack. See 
https://airflow.apache.org


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #8407: Fixes check_migrations commands

[GitBox]

kaxil commented on issue #8407: Fixes check_migrations commands
URL: https://github.com/apache/airflow/pull/8407#issuecomment-614727546
 
 
   I am surprised though that the CI didn't fail. Do we not have any tests for 
it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman opened a new pull request #8407: Fixes check_migrations commands

[GitBox]

dimberman opened a new pull request #8407: Fixes check_migrations commands
URL: https://github.com/apache/airflow/pull/8407
 
 
   Fixes bug where old "wait_for_migrations" function name
   is left in the lazy loader of the cli_parser
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] madison-ookla edited a comment on issue #8395: .airflowigore ignores files with "latest" in the name

[GitBox]

madison-ookla edited a comment on issue #8395: .airflowigore ignores files with 
"latest" in the name
URL: https://github.com/apache/airflow/issues/8395#issuecomment-614715043
 
 
   I'm willing to bet it's because `test` is matching in `latest` (la**test**)! 
If you have a test folder, try changing `test` in your `.airflowignore` to 
`test/` or making your test search more specific. 
   More info on how the `.airflowignore` works: 
https://airflow.apache.org/docs/stable/concepts.html#airflowignore


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #8405: Fix timing-based flakey test in TestLocalTaskJob

[GitBox]

ashb commented on issue #8405: Fix timing-based flakey test in TestLocalTaskJob
URL: https://github.com/apache/airflow/pull/8405#issuecomment-614715400
 
 
   There was on odd timeout/hang that caused one test job to fail _after_ 
pytest had finished running its tests.
   
   ```
   2020-04-16T14:32:00.8362178Z = 5500 passed, 224 skipped, 1 xfailed, 2 
xpassed, 13 warnings in 1660.97s (0:27:40) =
   2020-04-16T14:32:00.8362298Z No metrics to flush. Continuing.
   2020-04-16T14:32:00.8362409Z No distributions to flush. Continuing.
   2020-04-16T14:32:00.8362498Z No events to flush. Continuing.
   2020-04-16T14:32:00.9181541Z EOF in transport thread
   2020-04-16T14:32:00.9181725Z EOF in transport thread
   2020-04-16T14:32:00.9187980Z EOF in transport thread
   2020-04-16T15:09:29.6183593Z ##[error]The operation was canceled.
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj merged pull request #8404: Add colors to airflow config command

[GitBox]

mik-laj merged pull request #8404: Add colors to airflow config command
URL: https://github.com/apache/airflow/pull/8404
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] madison-ookla commented on issue #8395: .airflowigore ignores files with "latest" in the name

[GitBox]

madison-ookla commented on issue #8395: .airflowigore ignores files with 
"latest" in the name
URL: https://github.com/apache/airflow/issues/8395#issuecomment-614715043
 
 
   I'm willing to bet it's because `test` is matching in `latest` (la**test**)! 
If you have a test folder, try changing `test` to `test/` or making your test 
search more specific. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8405: Fix timing-based flakey test in TestLocalTaskJob

[GitBox]

ashb commented on a change in pull request #8405: Fix timing-based flakey test 
in TestLocalTaskJob
URL: https://github.com/apache/airflow/pull/8405#discussion_r409636383
 
 

 ##
 File path: tests/jobs/test_local_task_job.py
 ##
 @@ -325,28 +336,20 @@ def check_failure(context):
   session=session)
 ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
 ti.refresh_from_db()
+
 job1 = LocalTaskJob(task_instance=ti,
 ignore_ti_state=True,
 executor=SequentialExecutor())
-from airflow.task.task_runner.standard_task_runner import 
StandardTaskRunner
-job1.task_runner = StandardTaskRunner(job1)
-process = multiprocessing.Process(target=job1.run)
-process.start()
-ti.refresh_from_db()
-for _ in range(0, 50):
-if ti.state == State.RUNNING:
-break
-time.sleep(0.1)
-ti.refresh_from_db()
-self.assertEqual(State.RUNNING, ti.state)
-ti.state = State.FAILED
-session.merge(ti)
-session.commit()
+with timeout(30):
+# This should be _much_ shorter to run.
+# If you change this limit, make the timeout in the callbable 
above bigger
 
 Review comment:
   Yes, that. Sorry missed this comment before I hit merged.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb merged pull request #8405: Fix timing-based flakey test in TestLocalTaskJob

[GitBox]

ashb merged pull request #8405: Fix timing-based flakey test in TestLocalTaskJob
URL: https://github.com/apache/airflow/pull/8405
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8406: Unable to install apache-airflow 1.10.5

[GitBox]

boring-cyborg[bot] commented on issue #8406: Unable to install apache-airflow 
1.10.5 
URL: https://github.com/apache/airflow/issues/8406#issuecomment-614708686
 
 
   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] udhayakannan opened a new issue #8406: Unable to install apache-airflow 1.10.5

[GitBox]

udhayakannan opened a new issue #8406: Unable to install apache-airflow 1.10.5 
URL: https://github.com/apache/airflow/issues/8406
 
 
   I am seeing below error while downloading 1.10.5 version in ubuntu
Cache entry deserialization failed, entry ignored
 Downloading 
https://files.pythonhosted.org/packages/d5/12/6e5e4207483df7e82ac87079281f8f500e6610247ceb1464ed5592f4ceed/apache_airflow-1.10.5-py2.py3-none-any.whl
 (5.9MB)
   81% |██  | 4.8MB 4.2kB/s eta 
0:04:27Exception:
   Traceback (most recent call last):
 File 
"/usr/share/python-wheels/urllib3-1.22-py2.py3-none-any.whl/urllib3/response.py",
 line 302, in _error_catcher
   yield
 File 
"/usr/share/python-wheels/urllib3-1.22-py2.py3-none-any.whl/urllib3/response.py",
 line 384, in read
   data = self._fp.read(amt)
 File 
"/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/filewrapper.py",
 line 60, in readdata = self.__fp.read(amt)
 File "/usr/lib/python3.6/http/client.py", line 459, in read
   n = self.readinto(b)
 File "/usr/lib/python3.6/http/client.py", line 503, in readinto
   n = self.fp.readinto(b)
 File "/usr/lib/python3.6/socket.py", line 586, in readinto
   return self._sock.recv_into(b)
 File "/usr/lib/python3.6/ssl.py", line 1012, in recv_into
   return self.read(nbytes, buffer)
 File "/usr/lib/python3.6/ssl.py", line 874, in read
   return self._sslobj.read(len, buffer)
 File "/usr/lib/python3.6/ssl.py", line 631, in read
   v = self._sslobj.read(len, buffer)
   socket.timeout: The read operation timed out
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
 File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main
   status = self.run(options, args)
 File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 353, 
in run
   wb.build(autobuilding=True)
 File "/usr/lib/python3/dist-packages/pip/wheel.py", line 749, in build
   self.requirement_set.prepare_files(self.finder)
 File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 380, in 
prepare_files
   ignore_dependencies=self.ignore_dependencies))
 File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 620, in 
_prepare_file
   session=self.session, hashes=hashes)
 File "/usr/lib/python3/dist-packages/pip/download.py", line 821, in 
unpack_url
   hashes=hashes
 File "/usr/lib/python3/dist-packages/pip/download.py", line 659, in 
unpack_http_url
   hashes)
 File "/usr/lib/python3/dist-packages/pip/download.py", line 882, in 
_download_http_url
   _download_url(resp, link, content_file, hashes)
 File "/usr/lib/python3/dist-packages/pip/download.py", line 603, in 
_download_url
   hashes.check_against_chunks(downloaded_chunks)
 File "/usr/lib/python3/dist-packages/pip/utils/hashes.py", line 46, in 
check_against_chunks
   for chunk in chunks:
 File "/usr/lib/python3/dist-packages/pip/download.py", line 571, in 
written_chunks
   for chunk in chunks:
 File "/usr/lib/python3/dist-packages/pip/utils/ui.py", line 139, in iter
   for x in it:
 File "/usr/lib/python3/dist-packages/pip/download.py", line 560, in 
resp_read
   decode_content=False):
 File 
"/usr/share/python-wheels/urllib3-1.22-py2.py3-none-any.whl/urllib3/response.py",
 line 436, in stream
   data = self.read(amt=amt, decode_content=decode_content)
 File 
"/usr/share/python-wheels/urllib3-1.22-py2.py3-none-any.whl/urllib3/response.py",
 line 401, in read
   raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
 File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
   self.gen.throw(type, value, traceback)
 File 
"/usr/share/python-wheels/urllib3-1.22-py2.py3-none-any.whl/urllib3/response.py",
 line 307, in _error_catcher
   raise ReadTimeoutError(self._pool, None, 'Read timed out.')
   urllib3.exceptions.ReadTimeoutError: 
HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out
   
   1.10.10 is getting installed in ubuntu but unable to see the Airflow package 
in root folder.
   Please guide me


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8400: Less frequent dockerhub builds

[GitBox]

kaxil commented on a change in pull request #8400: Less frequent dockerhub 
builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409626652
 
 

 ##
 File path: scripts/ci/ci_build_dockerhub.sh
 ##
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ 
.CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ 
.*weeks.* ||
+  ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ 
.*years.* ||
+  ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   Does it show `09 hours` in the `docker image ls ${DOCKER_IMAGE_SPEC}" 
--format '{{ .CreatedSince }}'`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jcarless commented on a change in pull request #8304: [Airflow-5565] Add extra_args param to s3hook

[GitBox]

jcarless commented on a change in pull request #8304: [Airflow-5565] Add 
extra_args param to s3hook
URL: https://github.com/apache/airflow/pull/8304#discussion_r409626349
 
 

 ##
 File path: airflow/providers/amazon/aws/hooks/s3.py
 ##
 @@ -576,19 +578,21 @@ def load_file_obj(self,
 object to be uploaded
 :type acl_policy: str
 """
-self._upload_file_obj(file_obj, key, bucket_name, replace, encrypt, 
acl_policy)
+self._upload_file_obj(file_obj, key, bucket_name, replace, encrypt, 
acl_policy, extra_args)
 
 def _upload_file_obj(self,
  file_obj,
  key,
  bucket_name=None,
  replace=False,
  encrypt=False,
- acl_policy=None):
+ acl_policy=None,
+ extra_args=None):
 if not replace and self.check_for_key(key, bucket_name):
 raise ValueError("The key {key} already exists.".format(key=key))
 
-extra_args = {}
+if not extra_args:
 
 Review comment:
   @potiuk 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #8095: stop rendering some class docs in wrong place

[GitBox]

kaxil commented on issue #8095: stop rendering some class docs in wrong place
URL: https://github.com/apache/airflow/pull/8095#issuecomment-614703960
 
 
   > Ok @kaxil @turbaszek I'm up for making this change. It does have some 
ripple effects: one is that it requires minor changes to 
operators-and-hooks-ref.rst, which is no problem. Another is that 
test_project_structure fails because the new module does not have a 
corresponding test module. Question for the group: presumably 
YandexCloudBaseHook is covered by tests of 
airflow.providers.yandex.hooks.yandexcloud_dataproc.DataprocHook. Should I 
write a test suite anyway for YandexCloudBaseHook? Or should I add the new 
module to MISSING_TEST_FILES in tests/test_project_structure.py?
   
   Let's keep it in MISSING_TEST_FILES for now and we can revisit it later


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil edited a comment on issue #8095: stop rendering some class docs in wrong place

[GitBox]

kaxil edited a comment on issue #8095: stop rendering some class docs in wrong 
place
URL: https://github.com/apache/airflow/pull/8095#issuecomment-614703960
 
 
   > Ok @kaxil @turbaszek I'm up for making this change. It does have some 
ripple effects: one is that it requires minor changes to 
operators-and-hooks-ref.rst, which is no problem. Another is that 
test_project_structure fails because the new module does not have a 
corresponding test module. Question for the group: presumably 
YandexCloudBaseHook is covered by tests of 
airflow.providers.yandex.hooks.yandexcloud_dataproc.DataprocHook. Should I 
write a test suite anyway for YandexCloudBaseHook? Or should I add the new 
module to MISSING_TEST_FILES in tests/test_project_structure.py?
   
   Let's keep it in MISSING_TEST_FILES for now and we can revisit it later in a 
separate PR


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8405: Fix timing-based flakey test in TestLocalTaskJob

[GitBox]

kaxil commented on a change in pull request #8405: Fix timing-based flakey test 
in TestLocalTaskJob
URL: https://github.com/apache/airflow/pull/8405#discussion_r409619288
 
 

 ##
 File path: tests/jobs/test_local_task_job.py
 ##
 @@ -325,28 +336,20 @@ def check_failure(context):
   session=session)
 ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
 ti.refresh_from_db()
+
 job1 = LocalTaskJob(task_instance=ti,
 ignore_ti_state=True,
 executor=SequentialExecutor())
-from airflow.task.task_runner.standard_task_runner import 
StandardTaskRunner
-job1.task_runner = StandardTaskRunner(job1)
-process = multiprocessing.Process(target=job1.run)
-process.start()
-ti.refresh_from_db()
-for _ in range(0, 50):
-if ti.state == State.RUNNING:
-break
-time.sleep(0.1)
-ti.refresh_from_db()
-self.assertEqual(State.RUNNING, ti.state)
-ti.state = State.FAILED
-session.merge(ti)
-session.commit()
+with timeout(30):
+# This should be _much_ shorter to run.
+# If you change this limit, make the timeout in the callbable 
above bigger
 
 Review comment:
   Where is the other timeout in the callable? Sorry the comment is a bit 
unclear. Are you talking about `time.sleep(60)` above


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8405: Fix timing-based flakey test in TestLocalTaskJob

[GitBox]

kaxil commented on a change in pull request #8405: Fix timing-based flakey test 
in TestLocalTaskJob
URL: https://github.com/apache/airflow/pull/8405#discussion_r409616777
 
 

 ##
 File path: tests/jobs/test_local_task_job.py
 ##
 @@ -325,28 +336,20 @@ def check_failure(context):
   session=session)
 ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
 ti.refresh_from_db()
+
 job1 = LocalTaskJob(task_instance=ti,
 ignore_ti_state=True,
 executor=SequentialExecutor())
-from airflow.task.task_runner.standard_task_runner import 
StandardTaskRunner
-job1.task_runner = StandardTaskRunner(job1)
-process = multiprocessing.Process(target=job1.run)
-process.start()
-ti.refresh_from_db()
-for _ in range(0, 50):
-if ti.state == State.RUNNING:
-break
-time.sleep(0.1)
-ti.refresh_from_db()
-self.assertEqual(State.RUNNING, ti.state)
-ti.state = State.FAILED
-session.merge(ti)
-session.commit()
+with timeout(30):
+# This should be _much_ shorter to run.
+# If you change this limit, make the timeout in the callbable 
above bigger
 
 Review comment:
   ```suggestion
   # If you change this limit, make the timeout in the callable 
above bigger
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services