[
https://issues.apache.org/jira/browse/AIRFLOW-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned AIRFLOW-1171:
-------------------------------------
Assignee: Holden Karau's magical unicorn (was: Richard Lee)
> Encoding error for non latin-1 Postgres database
> ------------------------------------------------
>
> Key: AIRFLOW-1171
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1171
> Project: Apache Airflow
> Issue Type: Bug
> Components: db, hooks
> Affects Versions: Airflow 1.8
> Environment: macOS 10.12.5
> Python 2.7.12
> Postgres 9.6.1
> However, these are irrelevant to this issue.
> Reporter: Richard Lee
> Assignee: Holden Karau's magical unicorn
> Priority: Major
>
> There's [a known issue|https://github.com/psycopg/psycopg2/issues/331] from
> psycopg2 that Airflow ignores the encoding settings from db by default and
> which results in encoding error if there's any non latin-1 content in
> database cell.
> Reference stack trace:
> {code}
> File "dags/recipe_hourly_pageviews.py", line 73, in <module>
> dag.cli()
> File
> "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/models.py",
> line 3339, in cli
> args.func(args, self)
> File
> "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/bin/cli.py",
> line 585, in test
> ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
> File
> "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/utils/db.py",
> line 53, in wrapper
> result = func(*args, **kwargs)
> File
> "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/models.py",
> line 1374, in run
> result = task_copy.execute(context=context)
> File
> "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/operators/generic_transfer.py",
> l
> ine 78, in execute
> destination_hook.insert_rows(table=self.destination_table, rows=results)
> File
> "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/hooks/dbapi_hook.py",
> line 215, i
> n insert_rows
> l.append(self._serialize_cell(cell, conn))
> File
> "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/hooks/postgres_hook.py",
> line 70,
> in _serialize_cell
> return psycopg2.extensions.adapt(cell).getquoted().decode('utf-8')
> UnicodeEncodeError: 'latin-1' codec can't encode characters in position 6-10:
> ordinal not in range(256)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
