Hey Ash,
JSON-RPC came from using best practices on how we can encode "actions"
in JSON. Mateusz posted tresult of his POC at the beginning of this
thread.
Unlike standard REST (which is not very useful in our case) it has a
way to POST JSON formatted requests (no parameters passed via URL) and
receive JSON Formatted responses both of which are not referring to
resources but actions to execute).
JSON RPC defines standard for how those should be encoded:
- single endpoint to handle it
- method name ("action") to execute
- parameters to pass
- unique id (useful when you want to use multiplexing to match request
with response over single stream)
- result - including error code handling standards
https://www.jsonrpc.org/specification
It has no other semantics included. It just handles the approach
where we are not operating on Resources but execute actions (which is
our case).
J.
On Tue, Nov 8, 2022 at 5:27 PM Ash Berlin-Taylor <[email protected]> wrote:
>
> Where did JSON-RPC come from? I'd like to catch up to speed on the
> discussion/reason. It is a very... crufty protocol and not one I would ever
> personally choose for new work.
>
> (Sorry I've been out a bit and may have missed some context or discussion.)
>
> -ash
>
> On 8 November 2022 15:53:06 GMT, Jarek Potiuk <[email protected]> wrote:
>>
>> Very good point. I have not thought about it but this is a very strong
>> reason to use our JSONSerialization.
>>
>> J,
>>
>> On Tue, Nov 8, 2022 at 4:28 PM Mateusz Henc <[email protected]> wrote:
>>>
>>>
>>> I just learned from https://docs.python.org/3/library/pickle.html
>>> Warning The pickle module is not secure. Only unpickle data you trust.
>>> It is possible to construct malicious pickle data which will execute
>>> arbitrary code during unpickling. Never unpickle data that could have come
>>> from an untrusted source, or that could have been tampered with.
>>>
>>> So there we have a "trusted" component - Internal API, exposing an
>>> endpoint that can be called from Worker, so from any arbitrary code. Unless
>>> there are some ways to protect from this, it seems that JSON serialization
>>> is the only option there.
>>>
>>> Best regards,
>>> Mateusz Henc
>>>
>>>
>>> On Thu, Nov 3, 2022 at 9:45 AM Mateusz Henc <[email protected]> wrote:
>>>>
>>>>
>>>> Thank you Giorgio.
>>>>
>>>> TBH I've never heard about FlatBuffer, but I will take a look.
>>>> The big advantage I see for Pickle is seamless integration - no
>>>> additional conversion code required for developers, especially that we get
>>>> the arguments as a dictionary. Pickle handles it without any problem (at
>>>> least in my tests). If FlatBuffer offers a similar experience then we
>>>> definitely should consider it.
>>>> The other question is if we should introduce yet another dependency to
>>>> Airflow - the number of pypi packages is big anyway, which leads to many
>>>> problems when users want to install their customer packages (dependency
>>>> conflicts etc).
>>>>
>>>> Best regards,
>>>> Mateusz Henc
>>>>
>>>>
>>>> On Wed, Nov 2, 2022 at 6:59 PM Giorgio Zoppi <[email protected]>
>>>> wrote:
>>>>>
>>>>>
>>>>> Hello,
>>>>> this is something i'd like to work too in my spare time but some i'd
>>>>> rather use flatbuffers for the payload since its duality json/binary.
>>>>> Flatbuffers have the nice feature that they're able to parse JSON files
>>>>> that conform to a schema into FlatBuffer binary files, so you can have
>>>>> duality json -> binary_on_wire-> dataclasses at receiver side. I'd rather
>>>>> put pickle as last resort or avoid at all. @Mateus please feel free to
>>>>> sync with me privately on missing actions items to make this feature a
>>>>> success.
>>>>> Just 1c,
>>>>> Best Regads,
>>>>> Giorgio
>>>>>
