Thank you Giorgio. TBH I've never heard about FlatBuffer, but I will take a look. The big advantage I see for Pickle is seamless integration - no additional conversion code required for developers, especially that we get the arguments as a dictionary. Pickle handles it without any problem (at least in my tests). If FlatBuffer offers a similar experience then we definitely should consider it. The other question is if we should introduce yet another dependency to Airflow - the number of pypi packages is big anyway, which leads to many problems when users want to install their customer packages (dependency conflicts etc).
Best regards, Mateusz Henc On Wed, Nov 2, 2022 at 6:59 PM Giorgio Zoppi <[email protected]> wrote: > Hello, > this is something i'd like to work too in my spare time but some i'd > rather use flatbuffers for the payload since its duality json/binary. > Flatbuffers have the nice feature that they're able to parse JSON > <http://json.org> files that conform to a schema into FlatBuffer binary > files, so you can have duality json -> binary_on_wire-> dataclasses at > receiver side. I'd rather put pickle as last resort or avoid at all. > @Mateus please feel free to sync with me privately on missing actions items > to make this feature a success. > Just 1c, > Best Regads, > Giorgio > >
