Fokko commented on issue #4766: [AIRFLOW-3720] Add prefix to file match in GCS_TO_S3 operator to avoid missmatch URL: https://github.com/apache/airflow/pull/4766#issuecomment-477950830 Now the Flake8 is passing, but some actual tests are failing: ``` ====================================================================== 38) ERROR: test_execute (tests.contrib.operators.test_gcs_to_s3_operator.GoogleCloudStorageToS3OperatorTest) ---------------------------------------------------------------------- Traceback (most recent call last): .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/moto/core/models.py line 71 in wrapper result = func(*args, **kwargs) .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py line 1305 in patched return func(*args, **keywargs) tests/contrib/operators/test_gcs_to_s3_operator.py line 70 in test_execute uploaded_files = operator.execute(None) airflow/contrib/operators/gcs_to_s3.py line 125 in execute replace=self.replace) airflow/hooks/S3_hook.py line 413 in load_bytes raise ValueError("The key {key} already exists.".format(key=key)) ValueError: The key TEST1.csv already exists. -------------------- >> begin captured logging << -------------------- airflow.task.operators: INFO: Getting list of the files. Bucket: test-bucket; Delimiter: .csv; Prefix: TEST airflow.task.operators: INFO: Saving file to s3://bucket/TEST2.csv airflow.hooks.S3_hook.S3Hook: INFO: Not Found airflow.task.operators: INFO: Saving file to s3://bucket/TEST1.csv --------------------- >> end captured logging << --------------------- ====================================================================== 39) ERROR: test_execute_replace (tests.contrib.operators.test_gcs_to_s3_operator.GoogleCloudStorageToS3OperatorTest) ---------------------------------------------------------------------- Traceback (most recent call last): .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/moto/core/models.py line 71 in wrapper result = func(*args, **kwargs) .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/mock/mock.py line 1305 in patched return func(*args, **keywargs) tests/contrib/operators/test_gcs_to_s3_operator.py line 95 in test_execute_replace b.put_object(Key=MOCK_FILES, Body=b'testing') .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/boto3/resources/factory.py line 520 in do_action response = action(self, *args, **kwargs) .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/boto3/resources/action.py line 83 in __call__ response = getattr(parent.meta.client, operation_name)(**params) .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/botocore/client.py line 314 in _api_call return self._make_api_call(operation_name, kwargs) .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/botocore/client.py line 586 in _make_api_call api_params, operation_model, context=request_context) .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/botocore/client.py line 621 in _convert_to_request_dict api_params, operation_model) .tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/botocore/validate.py line 291 in serialize_to_request raise ParamValidationError(report=report.generate_report()) ParamValidationError: Parameter validation failed: Invalid type for parameter Key, value: ['TEST1.csv', 'TEST2.csv', 'TEST3.csv'], type: <type 'list'>, valid types: <type 'basestring'> ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
