potiuk commented on issue #16386: URL: https://github.com/apache/airflow/issues/16386#issuecomment-1060017364
> what's the solution here? I am running a SageMakerProcessingOperator Airflow task the processing job executed successfully but I got the error "ERROR - Could not serialize the XCom value into JSON. If you are using pickle instead of JSON for XCom, then you need to enable pickle support for XCom in your airflow config., TypeError: Object of type datetime is not JSON serializable" . How should I get away from this ? I have downstream task depend on success of SageMakerProcessingOperator a) set `do_xcom_push` to False: https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/baseoperator/index.html b) derive your own custom operator from the SageMarkerOperator and return nothing (or converte return value to serializable. Smth like: ``` class MyOperator(SageMakerProcessingOperator): def execute(): ret = super().execute() serializable_return = ..... # convert it to serializable return (serializable_return) ``` c) enable pickle in Airflow configuration https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html?highlight=pickle#enable-xcom-pickling d) correct it in the operator and contribute it back (this is by far the best idea). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
