I'm no security expert, but we've got a big pile of serialization code that is kind of like JSON-pickly, but it will only deserialize known objects. it's a bit of pain to declare what you want to work with, but it seems safer.
I also have a newer system (built on top of dataclasses) that serializes to JSON, but it's "pure" JSON -- it does not store any info about object types or anything in the JSON. Rather, the deserializer knows what it expects, and can only unpack that. e.g., you set a class attribute to the type "List_of_Object_A" -- and then it will use Object_A's deserializer to try to unpack the JSON. It does mean you can't "just save" an arbitrary object, but you can use a decorator on a datacalsss to make any object savable. I did it this way because I want the JSON to be plain old JSON, but I also think it's substantially more secure. part of a non-released code base, but iof folks think it would be handy, I can pull it out into its own package. -CHB On Mon, Jul 13, 2020 at 9:54 AM Wes Turner <wes.tur...@gmail.com> wrote: > Looks like pyro4 (python remote objects) has moved to the serpent library > (as.literal_eval) [1] > > > defaults to a safe serializer (serpent > https://pypi.python.org/pypi/serpent ) that supports many Python data > types. > > supports different serializers (serpent, json, marshal, msgpack, pickle, > cloudpickle, dill) > > And pyro5 has removed support for unsafe serializers [5]: > > > no support for unsafe serializers AT ALL (pickle, dill, cloudpickle) - > only safe serializers (serpent, marshal, json, msgpack) > > > > for now, requires msgpack to be installed as well as serpent. > > TBH, I'm not sure how much any of the serialization protocols have been > fuzzed for safety purposes if at all. > > [4] https://github.com/irmen/Pyro4#feature-overview > > [5] > https://pyro5.readthedocs.io/en/latest/intro.html#what-has-been-changed-since-pyro4 > > (FWIW, PyArrow has very fast support for "Streaming, Serialization, and > IPC" > https://arrow.apache.org/docs/python/ipc.html and support for "Arbitrary > Object Serialization" > https://arrow.apache.org/docs/python/ipc.html#arbitrary-object-serialization > w/ pyarrow.serialize() and pyarrow.deserialize()) > > On Mon, Jul 13, 2020, 7:57 AM Chris Angelico <ros...@gmail.com> wrote: > >> On Mon, Jul 13, 2020 at 8:58 PM Edwin Zimmerman <ed...@211mainstreet.net> >> wrote: >> > >> > On 7/11/2020 11:17 PM, Greg Ewing wrote: >> > >> > On 12/07/20 1:01 pm, Edwin Zimmerman wrote: >> > >> > As I see it, the unsafe callables (eval, exec, os.system, etc) are >> generally functions, and safe ones(int, list, dict) are generally classes, >> though there certainly would be exceptions. >> > >> > >> > Where security is concerned, "there certainly would be exceptions" >> > are not words you want to hear. >> > >> > Agreed, that is why pickle should almost never be used. In the past, I >> have looked long and hard at using pickle in my own projects, but was >> always turned away because of its potential for security issues. I've >> thought for years that pickle is a major security foot gun, and I think >> that not allowing this by default: >> > >>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.") >> > would be a step in the right direction. >> > >> >> A pickle file (or equivalent blob in a database, or whatever) should >> be considered equally as trusted as your source code. If you're >> writing out a file that has the exact same access permissions as your >> own source code, and then reading it back, you shouldn't have to worry >> about pickle's safety any more than you worry about your code's safety >> - anyone who could maliciously craft something for you to unpickle >> could equally just edit the source code directly. >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list -- python-ideas@python.org >> To unsubscribe send an email to python-ideas-le...@python.org >> https://mail.python.org/mailman3/lists/python-ideas.python.org/ >> Message archived at >> https://mail.python.org/archives/list/python-ideas@python.org/message/HRJVLRL4FDF4MQ6EPEHC36P6CHZSQBZ3/ >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > _______________________________________________ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/7CLM57BK3YRFOOZBSEIWV65EKPQLZXPJ/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/6272LNYLAOB3TNM36G2YBMSSNS3NHNHT/ Code of Conduct: http://python.org/psf/codeofconduct/
