Where did JSON-RPC come from? I'd like to catch up to speed on the discussion/reason. It is a very... crufty protocol and not one I would ever personally choose for new work.
(Sorry I've been out a bit and may have missed some context or discussion.) -ash On 8 November 2022 15:53:06 GMT, Jarek Potiuk <[email protected]> wrote: >Very good point. I have not thought about it but this is a very strong >reason to use our JSONSerialization. > >J, > >On Tue, Nov 8, 2022 at 4:28 PM Mateusz Henc <[email protected]> wrote: >> >> I just learned from https://docs.python.org/3/library/pickle.html >> Warning The pickle module is not secure. Only unpickle data you trust. >> It is possible to construct malicious pickle data which will execute >> arbitrary code during unpickling. Never unpickle data that could have come >> from an untrusted source, or that could have been tampered with. >> >> So there we have a "trusted" component - Internal API, exposing an endpoint >> that can be called from Worker, so from any arbitrary code. Unless there are >> some ways to protect from this, it seems that JSON serialization is the only >> option there. >> >> Best regards, >> Mateusz Henc >> >> >> On Thu, Nov 3, 2022 at 9:45 AM Mateusz Henc <[email protected]> wrote: >>> >>> Thank you Giorgio. >>> >>> TBH I've never heard about FlatBuffer, but I will take a look. >>> The big advantage I see for Pickle is seamless integration - no additional >>> conversion code required for developers, especially that we get the >>> arguments as a dictionary. Pickle handles it without any problem (at least >>> in my tests). If FlatBuffer offers a similar experience then we definitely >>> should consider it. >>> The other question is if we should introduce yet another dependency to >>> Airflow - the number of pypi packages is big anyway, which leads to many >>> problems when users want to install their customer packages (dependency >>> conflicts etc). >>> >>> Best regards, >>> Mateusz Henc >>> >>> >>> On Wed, Nov 2, 2022 at 6:59 PM Giorgio Zoppi <[email protected]> >>> wrote: >>>> >>>> Hello, >>>> this is something i'd like to work too in my spare time but some i'd >>>> rather use flatbuffers for the payload since its duality json/binary. >>>> Flatbuffers have the nice feature that they're able to parse JSON files >>>> that conform to a schema into FlatBuffer binary files, so you can have >>>> duality json -> binary_on_wire-> dataclasses at receiver side. I'd rather >>>> put pickle as last resort or avoid at all. @Mateus please feel free to >>>> sync with me privately on missing actions items to make this feature a >>>> success. >>>> Just 1c, >>>> Best Regads, >>>> Giorgio >>>>
