[GitHub] itscaro edited a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
itscaro edited a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push
flag for contrib's operators
URL:
https://github.com/apache/incubator-airflow/pull/3981#issuecomment-426179746
I got two errors and I could not find where they are coming from, if anyone
can have a look, it would be great. Curiously the labels is set to be None.
```==
23) FAIL: test_exec
(tests.contrib.operators.test_dataflow_operator.DataFlowPythonOperatorTest)
--
Traceback (most recent call last):
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py
line 1305 in patched
return func(*args, **keywargs)
tests/contrib/operators/test_dataflow_operator.py line 115 in test_exec
PY_OPTIONS)
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py
line 948 in assert_called_once_with
return self.assert_called_with(*args, **kwargs)
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py
line 937 in assert_called_with
six.raise_from(AssertionError(_error_message(cause)), cause)
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/six.py
line 737 in raise_from
raise value
AssertionError: Expected call:
start_python_dataflow('test-dataflow-pipeline', {'labels': {'airflow-version':
'v2-0-0-dev0-incubating', 'foo': 'bar'}, 'project': 'test', 'output':
'gs://test/output', 'staging_location': 'gs://test/staging'}, , ['-m'])
Actual call: start_python_dataflow('test-dataflow-pipeline', {None:
'gs://test/output'}, ,
['-m'])
==
24) FAIL: testSuccessfulRun
(tests.contrib.operators.test_mlengine_operator_utils.CreateEvaluateOpsTest)
--
Traceback (most recent call last):
tests/contrib/operators/test_mlengine_operator_utils.py line 125 in
testSuccessfulRun
['-m'])
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py
line 948 in assert_called_once_with
return self.assert_called_with(*args, **kwargs)
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py
line 937 in assert_called_with
six.raise_from(AssertionError(_error_message(cause)), cause)
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/six.py
line 737 in raise_from
raise value
AssertionError: Expected call: start_python_dataflow('{{task.task_id}}',
{'metric_keys': 'err', 'prediction_path': 'gs://legal-bucket/fake-output-path',
'metric_fn_encoded':
'gAJjZGlsbC5fZGlsbApfY3JlYXRlX2Z1bmN0aW9uCnEAKGNkaWxsLl9kaWxsCl9sb2FkX3R5cGUKcQFVCENvZGVUeXBlcQKFcQNScQQoSwFLAUsBSlNgAQBVBGQCAFNxBU5HP7mZmZmZmZpHP7mZmZmZmZqFcQaHcQcpVQF4cQiFcQlVPC9hcHAvdGVzdHMvY29udHJpYi9vcGVyYXRvcnMvdGVzdF9tbGVuZ2luZV9vcGVyYXRvcl91dGlscy5weXEKVQg8bGFtYmRhPnELS0lVAHEMKSl0cQ1ScQ59cQ9oC05OfXEQdHERUnESLg==',
'labels': {'airflow-version': 'v2-0-0-dev0-incubating'}},
'airflow.contrib.operators.mlengine_prediction_summary', ['-m'])
Actual call: start_python_dataflow('{{task.task_id}}', {None:
{'airflow-version': 'v2-0-0-dev0-incubating'}},
'airflow.contrib.operators.mlengine_prediction_summary', ['-m'])
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] itscaro edited a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
itscaro edited a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#issuecomment-425882500 @ashb I will do the change for other operators in this PR. :) --- I updated other operators in a new commit. There is a special case with `GoogleCloudStorageToBigQueryOperator` which uses `max_id_key` as a flag to enable `xcom_push` feature, to harmonize argument between operators I added `do_xcom_push` as well and documented that `max_id_key` is used in combination with `do_xcom_push` Between `xcom_push` and `do_xcom_push`, should I rename to `xcom_push` for argument & `self.xcom_push_flag` for property as in base operators? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] itscaro edited a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
itscaro edited a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#issuecomment-425882500 @ashb I will do the change for other operators in this PR. :) --- I updated other operators in a new commit. There is a special case with `GoogleCloudStorageToBigQueryOperator` which uses `max_id_key` as a flag for `xcom_push`, to harmonize argument between operators I added `do_xcom_push` as well and documented that `max_id_key` is used in combination with `do_xcom_push` Between `xcom_push` and `do_xcom_push`, should I rename to `xcom_push` for argument & `xcom_push_flag` for property as in base operators? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] itscaro edited a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
itscaro edited a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#issuecomment-425882500 @ashb I will do the change for other operators in this PR. :) --- I updated other operators in a new commit. There is a special case with `GoogleCloudStorageToBigQueryOperator` which uses `max_id_key` as a flag for `xcom_push`, to harmonize argument between operators I added `do_xcom_push` as well and documented that `max_id_key` is used in combination with `do_xcom_push` Between `xcom_push` and `do_xcom_push`, should I rename to `xcom_push` for argument & `self.xcom_push_flag` for property as in base operators? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] itscaro edited a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
itscaro edited a comment on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#issuecomment-425882500 @ashb I will do the change for other operators in this PR. :) --- I updated other operators in a new commit. There is a special case with `GoogleCloudStorageToBigQueryOperator` which uses `max_id_key` as a flag for `xcom_push`, to harmonize argument between operators I added `do_xcom_push` as well and documented that `max_id_key` is used in combination with `do_xcom_push` Between `xcom_push` and `do_xcom_push`, should I rename to `xcom_push` as in base operators? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
