Does the specification provide tests for us to include? Otherwise we can include enough tests for full line coverage and a “brute force”/property test commented out.
I would say the name “replace_invalid” is excellent. On Tue, Oct 31, 2023 at 18:52 Cameron Duley <cameron.dule...@gmail.com> wrote: > This was the final version I'd landed on for UTF-8: > > https://github.com/elixir-unicode/unicode/blob/main/lib/unicode/validation/utf8.ex > > Along with the following modules for testing: > > https://github.com/elixir-unicode/unicode/blob/main/test/support/unicode_validation_helpers.ex > > https://github.com/elixir-unicode/unicode/blob/main/test/unicode_validation_test.exs > > I think it's ideal functionality to have in the String module, and the > implementation's "reasonable enough" until a native solution's available in > OTP. > > Testing is my only uncertainty - How much is prudent and in what manner? > On Tuesday, October 31, 2023 at 12:35:48 PM UTC-4 José Valim wrote: > >> Folks, I am following up on this, where did we land? >> >> The new implementation is roughly ~70LOC for UTF-8, so at first I don't >> see an issue with adding it to Elixir. However, the Elixir version would be >> UTF-8 only (part of the String module). >> >> Thoughts? >> >> On Saturday, October 7, 2023 at 10:51:34 PM UTC+2 Kip wrote: >> >>> Cameron, I think this would be a useful addition to the Unicode library >>> <https://github.com/elixir-unicode/unicode> I maintain. If that works >>> for you, please open an issue there and we can collaborate. I think it >>> being part of the Erlang `:unicode` module makes good sense too as José >>> says but that's a longer "sales" and implementation cycle. >>> >>> On Saturday, October 7, 2023 at 7:40:50 PM UTC+11 José Valim wrote: >>> >>>> Hi Cameron, >>>> >>>> If the goal is to include this handling for UTF-16 and UTF-32, I >>>> suggest proposing this to Erlang/OTP as new functions in the "unicode" >>>> module. Otherwise, Elixir only has facilities to deal with UTF-8. You could >>>> propose such a feature in their issues tracker. >>>> >>>> Also note that "rolling your own" or "depending on packages" is usually >>>> not enough reasons for adding features to Elixir. Otherwise, one could >>>> easily argue Decimal and Jason would be more important additions to the >>>> language. :) We do describe which features we would consider part of the >>>> language here: https://elixir-lang.org/development.html >>>> >>>> Other than that, awesome job on the library and benchmarks. :) >>>> >>>> On Sat, Oct 7, 2023 at 1:03 AM Kip <kipc...@gmail.com> wrote: >>>> >>>>> Your implementation is definitely fast and memory efficient so I >>>>> retract my implementation comments. Now that I've run the benchmarking >>>>> script and tested out a few different approaches leveraging the std lib I >>>>> understand better why you've taken the approach you have. Nice work. >>>>> >>>>> On Saturday, October 7, 2023 at 9:26:37 AM UTC+11 Kip wrote: >>>>> >>>>>> Cameron, I think this is a useful proposal. Elixir has means to >>>>>> check validity (String.valid?/1) and a mechanism to split valid and >>>>>> invalid >>>>>> code points (String.chunk/2 with the :valid trait). But there isn't, to >>>>>> my >>>>>> knowledge, a means to coerce validity. A couple of thoughts: >>>>>> >>>>>> 1. Since Elixir strings are, by definition, UTF8, I don't know that >>>>>> special handling of UTF16 and UTF32 code points makes much sense - >>>>>> although >>>>>> I accept this may be more Unicode compliant. >>>>>> 2. What would the function be called? Since we have String.valid?/1 >>>>>> maybe String.validate/2 with an option `replace_invalid: utf8_string`. >>>>>> The >>>>>> default `:replace_invalid` could be U+FFFD or it could be `nil`. >>>>>> If the default is `nil` then there could also be a `String.validate!/2` >>>>>> that raises if there is no `:replace_invalid` option. >>>>>> 3. I think the implementation could leverage the code of >>>>>> `String.chunk/2` which uses `String.next_codepoint/1`. That would >>>>>> simplify >>>>>> implementation and be more consistent in code style. >>>>>> >>>>>> On Friday, October 6, 2023 at 12:24:28 PM UTC+11 cameron...@gmail.com >>>>>> wrote: >>>>>> >>>>>>> As far as I can tell, neither Elixir nor Erlang have a built in >>>>>>> function for replacing invalid sequences in Unicode. There's a suggested >>>>>>> method on this page >>>>>>> <https://www.unicode.org/versions/Unicode15.0.0/UnicodeStandard-15.0.pdf#page=153> >>>>>>> of the Unicode standard for handling this. Several other languages ( >>>>>>> Go <https://pkg.go.dev/bytes#ToValidUTF8>, Python >>>>>>> <https://docs.python.org/3/library/stdtypes.html#bytes.decode>, C# >>>>>>> <https://github.com/dotnet/docs/issues/13547>, etc) now follow this >>>>>>> spec. >>>>>>> >>>>>>> Invalid Unicode's encountered frequently enough that I think it's >>>>>>> worth incorporating a solution into Elixir itself. >>>>>>> >>>>>>> Present alternatives to handling invalid unicode (and json by >>>>>>> extension <https://github.com/michalmuskala/jason/issues/174>) are: >>>>>>> >>>>>>> - Crashing (not ideal in many cases) >>>>>>> - Roll your own (lot of overhead for accidental complexity) >>>>>>> - Depend on a package (+1 package towards dependency hell) >>>>>>> >>>>>>> This is my college try >>>>>>> <https://github.com/Moosieus/UniRecover/tree/main>, but I'm certain >>>>>>> there's a performant and far cleaner solution to be had in pure Elixir. >>>>>>> If >>>>>>> not, perhaps this is a request for OTP. >>>>>>> >>>>>> >>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elixir-lang-core" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to elixir-lang-co...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elixir-lang-core/197620a2-6a96-41c6-a6e7-5da03e351080n%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/elixir-lang-core/197620a2-6a96-41c6-a6e7-5da03e351080n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elixir-lang-core+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elixir-lang-core/26e4c761-caf0-46ea-ab08-a33407febdb8n%40googlegroups.com > <https://groups.google.com/d/msgid/elixir-lang-core/26e4c761-caf0-46ea-ab08-a33407febdb8n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LKEOb%3D7WMeUhots27dj%3DAwvhLF3o%2Bo7Q2%2BhZXMNQi8YA%40mail.gmail.com.
