Under what conditions would one want to use pickling right now?
On Fri, Jul 22, 2016 at 9:43 AM, Maxime Beauchemin <[email protected]> wrote: > Pickling is kind of a dead feature, the original idea was that it would be > used to version and ship DAG definitions around, but in practice many > important things are not pickleable (jinja template objects, whatever may > be in callbacks or attached to a DAG object). Also I believe that the > MesosExecutor relies on pickling. Note that there is a hack to pickle the > template, but it has shortcomings (the pickled template cannot reference > other files in `includes` or `extends` calls) > > We should phase pickles out, probably by parameterizing the behavior in the > current version (generate_pickles=True), set the default to true (to match > the current behavior), but warn that this won't be an option in 2.0, and to > set the option to False in order to prepare for the 2.0 migration. > > Max > > On Fri, Jul 22, 2016 at 2:31 AM, Bolke de Bruin <[email protected]> wrote: > >> Hi, >> >> Although some efforts are underway to allow for syncing of DAGs by git or >> something else, I was wondering why we actually pickle? As we are >> supporting none pickled DAGs as well there seems to be no need for >> serialization, ie. keeping the state of the DAG. As DAG definitions are >> relatively small, why don’t we send the whole DAG? Or am I completely >> overlooking something? >> >> Cheers >> Bolke
