Thanks Jeremiah for your response! After discussion with experienced airflow contributors. One gently way is, not deprecate it, but has a conf setting to show if use Pickle Type. Let’s say if the conf says does not use Pickle Type, before the object to be pickled, it can first be parsed as JSON. And if retrieve from Pickle Type column, it could recover from JSON as well.
Thanks, Rui Wang On 2017-02-19 09:37 (-0800), Jeremiah Lowin <[email protected]> wrote: > Rui,> > > Thanks for pointing this out, it's a valid concern.> > > I personally have no issue with swapping Pickle -> JSON, but there may be> > many Airflow users relying on the current behavior and I don't want to> > invalidate their DAGs with a PR.> > > On the other hand, I'm not sure of a way to "gently" deprecate the> > PickleType. Perhaps step 1 is to check if an XCom can be JSON serialized> > and if it can't, print a warning? Then step 2 is to enforce JSON> > serialization at a future date.> > > Any suggestions of how to implement this?> > > J> > > On Sat, Feb 18, 2017 at 10:16 AM Rui Wang <[email protected]>> > wrote:> > > > Hi,> > >> > > I created an JIRA issue: https://issues.apache.org/jira/browse/AIRFLOW-855> > > .> > >> > >> > > The JIRA task above gives pretty rich context. Briefly speaking, > > PickleType> > > gives the possible that run code/command on remote machines. This type can> > > serialize objects, which is a wide scope. I am wondering what kind of use> > > cases you have for using Xcom and its PickleType. If the use cases show > > the> > > possibility that replacing PickleType with JSON type, the probably this> > > security issue can be solved by using JSON type instead,> > >> > >> > > Thanks,> > > Rui Wang> > >> >
