One possibility to allow this and other features waiting on `persistent_term` already on OTP 21.0+ would be to provide a non-NIF fallback implementation. After all, `persistent_term is based on userspace-libraries written in Elixir and Erlang that were released before (c.f. the FastGlobal Elixir library on Hex and mochiglobal <https://github.com/mochi/mochiweb/blob/master/src/mochiglobal.erl> (part of the mochiweb Erlang library).)
Very nice to see steps being made to support inspect redaction! ~Marten On 06-11-2020 15:30, José Valim wrote: > After a brief discussion with the Elixir team, we are inclined to go > more with option 2. However, in order to make it efficient, we want to > use persistent_term, which is only available on 21.3 and Elixir > currently supports 21.0+. > > We will drop Erlang/OTP 21 support once Erlang/OTP 24 is out but it is > unclear if that will be before Elixir v1.12. Alternatively we can > require Erlang/OTP 21.3. In any case, I have added an item to track > this here: https://github.com/elixir-lang/elixir/issues/8414 > > Thank you! > > On Fri, Nov 6, 2020 at 10:25 AM Allen Wyma <[email protected] > <mailto:[email protected]>> wrote: > > Looks like they just use the derive functionality from inspect: > "@derive{Inspect, except:@ecto_redact_fields}" so you can also > look into using that. If it's for struct outside of your control, > that may be more difficult > > On Fri, Nov 6, 2020 at 5:21 PM 'Jayson Vantuyl' via > elixir-lang-core <[email protected] > <mailto:[email protected]>> wrote: > > That would help with Ecto. Unfortunately we still see a bunch > of stuff from the likes of Elixir GRPC and various structs > from libraries that don't have that kind of functionality. We > were kind of hoping for a top-down, whitelist approach. We > figure that it's easier to plug the leak from the top > (*inspect/2* itself) than it is to try to deal with all of the > leaks at the bottom (everything that tries to use it). > > On Friday, November 6, 2020 at 1:15:37 AM UTC-8 > [email protected] <mailto:[email protected]> wrote: > > I believe that the new changeset "redact" attribute should > help with this? > > On Fri, Nov 6, 2020 at 4:56 PM 'Jayson Vantuyl' via > elixir-lang-core <[email protected]> wrote: > > Hey everybody, > > _*Use Case*_ > > I have what I think is actually a pretty common > problem. All over my code base, there are uses of > *inspect/2*. This is a wonderful thing that helps > with debugging. It is less wonderful when it spews > PII all over your logs / error pages and you find > yourself having sent somebody's social security number > to DataDog or disclosed their home address on a 500 > page. Then your lawyer starts doing that thing where > their eye twitches, you need to notify four different > regulators on three continents, alert all of your > customers with scary messages, are made to attend > 3-hours of mandatory re-education wherein you learn to > recite the GDPR from memory, and eventually sacrifice > three interns to appease the compliance gods. > > _*Temporary Hack*_ > > What myself and some co-conspirators have done to > address this is overriding the *Inspect* protocol for > the built-in types to redact things by default and > then have a whitelist for the bits we want to show. > Given that our top-level logging metadata is a map, we > can pretty much just whitelist keys there and call it > a day. This works fabulously, but it violates > Erlang's expectations significantly. While Erlang is > probably used to that, it gets quite cross and refuses > to generate a release because it doesn't have a good > way to know which .beam file to put in the release. > > As you might imagine, I'm pretty bummed that I can't > use releases and have to ignore tons of very alarming > sounding warnings about redefining modules. > > _*Options*_ > > Could we consider some solutions to make redaction > require less unforgivable sins against code loading? > To start, three directions have been proposed by the > various people I've talked to: > > 1. Instead of implementing *Inspect* for the built-in > types, do that inspection in a handler for *Any*; > thereby allowing overriding of the built-in types > easily. > 2. Wedge something into a common entry point (maybe > in *Inspect.Algebra*?) allows us to specify a > global redaction function. Perhaps configure this > with a global config value? > 3. Implement some sort of overriding layer for just > the *Inspect* protocol. > > > In terms of pros and cons, for #1... > > * *Pro:* Works well for built-ins. > * *Pro:* Implementing this is very straightforward. > * *Pro:* This probably doesn't break any existing > code, very small blast radius. > * *Con:* Doesn't work at all as soon as anything > defines its own inspection protocol. > * *Con:* Isn't amenable to configuration at runtime > (maybe this is not an issue?). > > As for #2... > > * *Pro:* /Can/ be configured at runtime. > * *Pro:* I have no idea how to implement this and > *Inspect.Algebra* scares me. > * *Pro:* This probably doesn't break any existing > code, very small blast radius. > * *Con:* Given that the Inspect protocol is pretty > much "turn X into string", I'm not sure how much > redaction we could really do if we allow the > existing protocol to run. > > As for #3... > > * *Pro:* This provides a clear way to just replace > the protocol for a given type. > * *Pro:* Implementing this is very straightforward. > * **Pro:* *This probably doesn't break any existing > code, very small blast radius. > * *Con:* It's all fun and games until a library does > it, then you need Override2 or 3 or 4... > * *Con:* Probably gets redundant if there ever is a > blessed way to override protocols. > * *Con:* I already pitched this to a few Elixir > celebrities and they thought it was a bit too hacky. > > _*In Closing*_ > > So, yeah, in the long term, maybe we'll have a blessed > way to override protocols; but, short of that, there's > got to be some stopgap, right? What do people think > of adopting something like one of these approaches so > that my PII problems evaporate and I can finally build > some sweet, sweet releases? I'll even implement it > myself! I promise! > > Thanks, > > - J > > -- > You received this message because you are subscribed > to the Google Groups "elixir-lang-core" group. > To unsubscribe from this group and stop receiving > emails from it, send an email to > [email protected]. > To view this discussion on the web visit > > https://groups.google.com/d/msgid/elixir-lang-core/d5d9a932-930c-46f7-a3fb-2f1a9ae06112n%40googlegroups.com > > <https://groups.google.com/d/msgid/elixir-lang-core/d5d9a932-930c-46f7-a3fb-2f1a9ae06112n%40googlegroups.com?utm_medium=email&utm_source=footer>. > > -- > You received this message because you are subscribed to the > Google Groups "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from > it, send an email to > [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > > https://groups.google.com/d/msgid/elixir-lang-core/38307a23-2568-4c48-ae8c-76cb70e7541bn%40googlegroups.com > > <https://groups.google.com/d/msgid/elixir-lang-core/38307a23-2568-4c48-ae8c-76cb70e7541bn%40googlegroups.com?utm_medium=email&utm_source=footer>. > > -- > You received this message because you are subscribed to the Google > Groups "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, > send an email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > > https://groups.google.com/d/msgid/elixir-lang-core/CAJuWSyL8xhXbZf0ca_nZ4-dJU_Xej3UJ_8-dQFJp0%2B9XX1znpw%40mail.gmail.com > > <https://groups.google.com/d/msgid/elixir-lang-core/CAJuWSyL8xhXbZf0ca_nZ4-dJU_Xej3UJ_8-dQFJp0%2B9XX1znpw%40mail.gmail.com?utm_medium=email&utm_source=footer>. > > -- > You received this message because you are subscribed to the Google > Groups "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4L_CuMDq2JVLz2tYog33eTBLR_YF36B-c_DY5VUaVQNsQ%40mail.gmail.com > <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4L_CuMDq2JVLz2tYog33eTBLR_YF36B-c_DY5VUaVQNsQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/347b00be-fb63-5624-9e10-6d16bf4f1d79%40resilia.nl.
signature.asc
Description: OpenPGP digital signature
