Ben Chen created AIRFLOW-4363:
---------------------------------
Summary: Encounter JSON Decode Error when using docker operator
Key: AIRFLOW-4363
URL: https://issues.apache.org/jira/browse/AIRFLOW-4363
Project: Apache Airflow
Issue Type: Bug
Environment: - Mac OS 10.14
- python 3.6.8
- airflow 1.10.2
Reporter: Ben Chen
Assignee: Ben Chen
## Description
When using the docker_operator, I experienced some issue while using Mac OS
10.14.4. The error was json.JSONDecodeError. After my investigation about this
error, I found that there are several messages for logging aren't well
separated, for example it contains \n inside one single message which should be
split into 2 to more different messages.
[Update]
Confirmed that issue came from the implementation in airflow, issue cannot be
solved by just passing `decode` to parameter in docker.pull method in docker
api.
[Solution]
For now, I use try-catch to run the original implementation, and in the
exception part I split the message to list and then parse it. Looking for
simpler solution to this non critical but still blocking point.
## Logs
{docker_operator.py:188} INFO - Starting docker container from image hello-world
{docker_operator.py:202} INFO - Pulling docker image hello-world
{docker_operator.py:207} INFO - Pulling from library/hello-world
{docker_operator.py:207} INFO - Pulling fs layer
{docker_operator.py:207} INFO - Downloading
{docker_operator.py:207} INFO - Downloading
{docker_operator.py:207} INFO - Download complete
{docker_operator.py:207} INFO - Extracting
{docker_operator.py:207} INFO - Extracting
{docker_operator.py:207} INFO - Pull complete
{docker_operator.py:207} INFO - Digest:
sha256:92695bc579f31df7a63da6922075d0666e565ceccad16b59c3374d2cf4e8e50e
{docker_operator.py:207} INFO - Pulling from library/hello-world
{docker_operator.py:207} INFO - Digest:
sha256:1a67c1115b199aa9d964d5da5646917cbac2d5450c71a1deed7b1bfb79c2c82d
{models.py:1788} ERROR - Extra data: line 2 column 1 (char 70)
Traceback (most recent call last):
line 1657, in _run_raw_task, result = task_copy.execute(context=context)
line 205, in execute output = json.loads(line)
line 354, in loads, return _default_decoder.decode(s)
line 342, in decode, raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 70)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
