[
https://issues.apache.org/jira/browse/AIRFLOW-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16730853#comment-16730853
]
ASF GitHub Bot commented on AIRFLOW-2939:
-----------------------------------------
kaxil commented on pull request #4371: [AIRFLOW-2939][AIRFLOW-3568] fix
TypeError on GoogleCloudStorageToS3Operator / S3ToGoogleCloudStorageOperator
URL: https://github.com/apache/incubator-airflow/pull/4371
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> `set` fails in case of `exisiting_files is None` and in case of `json.dumps`
> ----------------------------------------------------------------------------
>
> Key: AIRFLOW-2939
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2939
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
> Affects Versions: 2.0.0
> Reporter: Kiyoshi Nomo
> Assignee: Yohei Onishi
> Priority: Major
> Fix For: 1.10.2
>
>
> h1. Problems
> h2. TypeError: 'NoneType' object is not iterable
> [https://github.com/apache/incubator-airflow/blob/06b62c42b0b55ea55b86b130317594738d2f36a2/airflow/contrib/operators/gcs_to_s3.py#L91]
>
> {code:java}
> >>> set(None)
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> TypeError: 'NoneType' object is not iterable
> {code}
>
> h2. TypeError: set(['a']) is not JSON serializable
> [https://github.com/apache/incubator-airflow/blob/b78c7fb8512f7a40f58b46530e9b3d5562fe84ea/airflow/models.py#L4483]
>
> {code:python}
> >>> json.dumps(set(['a']))
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/__init__.py",
> line 244, in dumps
> return _default_encoder.encode(obj)
> File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/encoder.py",
> line 207, in encode
> chunks = self.iterencode(o, _one_shot=True)
> File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/encoder.py",
> line 270, in iterencode
> return _iterencode(o, 0)
> File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/encoder.py",
> line 184, in default
> raise TypeError(repr(o) + " is not JSON serializable")
> TypeError: set(['a']) is not JSON serializable
> {code}
>
> h1. Solution
> * Check that the existing fils is not None.
> * Convert it to the `set` and return it to the `list` after get to the
> difference of files.
> {code:python}
> if existing_files is not None:
> files = list(set(files) - set(existing_files))
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
