Serialising XCom via pickle is being deprecated for Airflow 2 btw: https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;a=commitdiff;h=4cf904c;hp=984a87c0cb685ea4dfa765cc4f4a23c9058b3965 <https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;a=commitdiff;h=4cf904c;hp=984a87c0cb685ea4dfa765cc4f4a23c9058b3965>
> +- The pickle type for XCom messages has been replaced by json to prevent RCE > attacks. > + Note that JSON serialization is stricter than pickling, so if you want to > e.g. pass > + raw bytes through XCom you must encode them using an encoding like base64. > + By default pickling is still enabled until Airflow 2.0. To disable it > + Set enable_xcom_pickling = False in your Airflow config. Was your test performed against the release candidate or master? I thought the timezone-aware date change was only in the later. -ash > On 6 Dec 2017, at 15:35, Boris Tyukin <[email protected]> wrote: > > hm...can you elaborate why we cannot use "from datetime import datetime". > it worked before just fine :) i was under an impression that any pickle > serializable type can be using with xcom. > > Along with a recent change in logs (logging.info does not write to log > files anymore), this makes upgrade to 1.9 quite difficult. > > On Wed, Dec 6, 2017 at 9:19 AM, Bolke de Bruin <[email protected]> wrote: > >> You should not use a naive datetime as it mentioned. So instead of of >> using “from datetime import datetime” use “from airflow.utils.timezone >> import datetime”. This sets the timezone information that is required to >> the default configured in airflow.cfg. >> >> A patch that sets the default timezone for TaskInstance is also helpful in >> this case and will be welcomed. How to do it you can find in the DAG class. >> >> Bolke. >> >>> On 6 Dec 2017, at 15:13, Sumit Maheshwari <[email protected]> >> wrote: >>> >>> Folks, >>> >>> I've some testes, which used to pass earlier, but recently they started >> failing. When I looked closely it appeared that xcom_push is behaving >> differently. >>> >>> >>> So this is is the current code, which throws the error mentioned below >> to it >>> DEFAULT_DATE = datetime(2017, 1, 1) >>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE) >>> ti.xcom_push('abcd', 12345) >>> >>> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 31, >> in process_bind_param >>> raise ValueError('naive datetime is disallowed') >>> StatementError: (exceptions.ValueError) naive datetime is disallowed >> [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ? >> AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]] >>> >>> When I changed the execution date param to a string, it fails with other >> error. >>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE.isoformat()) >>> ti.xcom_push('abcd', 12345) >>> >>> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 29, >> in process_bind_param >>> repr(value)) >>> StatementError: (exceptions.TypeError) expected datetime.datetime, not >> '2017-01-01T00:00:00' [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND >> xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?'] >> [parameters: [{}]] >>> >>> Somehow I feel that it is linked with sqlalchemy_utc library, though I >> may be wrong and its just missing some timezone related settings. >>> Please let me know if anyone else has faced this issue as well. >>> >>> Thanks, >>> Sumit >>> >>> >> >>
