Hey Julien. Thanks, OK... 📖
The Python docs have it <https://docs.python.org/3.10/library/codecs.html#python-specific-encodings> : > If you need the IDNA 2008 standard from *RFC 5891* <https://tools.ietf.org/html/rfc5891.html> and *RFC 5895* <https://tools.ietf.org/html/rfc5895.html>, use the third-party idna module <https://pypi.org/project/idna/>. So the question is do we **need** the newer standard? I will have a read of the various resources here, and I'll also ask the Django Security Team if they have any thoughts. Kind Regards, Carlton On Tue, 6 Sept 2022 at 23:03, 'Julien Bernard' via Django developers (Contributions to Django itself) <django-developers@googlegroups.com> wrote: > Hi Carlton, > > IDNA 2008 made some changes in the valid or invalid IDNs and some > differences in the ways some characters are transformed in Punycode > compared to IDNA 2003 for multiple reasons. > A difference that is often used as an example is the german 'ß' character. > In IDNA 2003 it is transformed into 'ss' while it is converted into > Punycode in IDNA 2008. > It means that, depending on the standard that is implemented, you may > reach totally different domains with the same IDN, which may lead to > security issues. > For example, the URL https://fuß.standcore.com/ > <https://fuss.standcore.com/> would be https://fuss.standcore.com/ with > IDNA 2003 and https://xn--fu-hia.standcore.com/ with IDNA 2008. > This is only a very brief insight, for further quick readings, > https://www.unicode.org/faq/idn.html is quite informative too. > > Best regards, > Julien > > Le mardi 6 septembre 2022 à 14:39:49 UTC-4, carlton...@gmail.com a écrit : > >> Hey Julian. >> >> What's maybe missing is some concrete cases. "This conversion should be >> made IDNA 2008 compliant." — OK, but what does that buy us? >> >> Maybe the idna package is OK... It's widely depended on already — I got >> it for free yesterday installing httpx in a project — and packaging isn't >> what it was... — but **if** we take on an extra dependency it needs to be >> for a clear gain. >> >> Likely (still) a proof-of-concept at least showing what's added (as a >> separate package? 🤔) is the easiest way forward? >> >> Others may yet agree that this is something needed. >> >> Kind Regards, >> >> Carlton >> >> On Tue, 6 Sept 2022 at 20:23, 'Julien Bernard' via Django developers >> (Contributions to Django itself) <django-d...@googlegroups.com> wrote: >> >>> Thanks Carlton. >>> >>> This makes total sense to keep things simple and avoid bringing another >>> dependency in the context of validation, providing that you won't prevent >>> valid URLs to be accepted. That's where it can be tricky but it seems >>> reasonable to think that the current domain validation is too permissive >>> from what I saw. >>> But the punycode method is also used in other places where it is more >>> "critical" than a validator: >>> >>> - Urlizer >>> - CachedDnsName >>> - email sending. >>> >>> The need for the idna package should be evaluated regarding those usages >>> rather than the validation. >>> >>> Best regards, >>> Julien >>> >>> Le dimanche 4 septembre 2022 à 05:16:19 UTC-4, carlton...@gmail.com a >>> écrit : >>> >>>> Hi Julian. >>>> >>>> We've had several tickets and discussions surrounding how far e.g. >>>> URLValidator needs to match all valid URLs. The conclusion we've come to >>>> (which is always provisional) is that actually we **don't** want such match >>>> all valid (according to the relevant RFC and such) input. Rather, we'd >>>> prefer a simpler implementation — that's realistically maintainable for us >>>> — that captures the 95% case, and then ask users to implement a custom >>>> validator if they need more. This seems like a happy compromise. >>>> >>>> I hope that makes sense. >>>> >>>> I agree with Adam here that a third-party package is the immediate way >>>> forward here. If it shows a lot of demand then it's always open to revisit >>>> whether that should be included in Django itself. (Taking on an extra >>>> dependency for a small subset of users is always going to be up for debate >>>> — it's not a blocker per se, but it does need weighing...) >>>> >>>> Kind Regards, >>>> >>>> Carlton >>>> >>>> On Fri, 2 Sept 2022 at 17:32, 'Julien Bernard' via Django developers >>>> (Contributions to Django itself) <django-d...@googlegroups.com> wrote: >>>> >>>>> Thanks Adam. >>>>> The ticket was targetting EmailValidator but the punycode method is >>>>> used at more places in Django core. >>>>> If you look for Unicode characters support in email addresses in >>>>> tickets you will get more than looking for IDN but indeed the demand may >>>>> not be that much. >>>>> This is a vicious circle, IDNs and especially EAIs are not widely used >>>>> because of the poor support in software and the lack of compliance with >>>>> the >>>>> latest version. >>>>> Regards, >>>>> Julien >>>>> >>>>> >>>>> Le jeudi 1 septembre 2022 à 13:33:29 UTC-4, Adam Johnson a écrit : >>>>> >>>>>> Some data... >>>>>> >>>>>> The idna package has ~9M downloads a day ( >>>>>> https://pypistats.org/packages/idna ) compared with Django's ~350k ( >>>>>> https://pypistats.org/packages/django ). >>>>>> >>>>>> However it has 191 GitHub stars ( https://github.com/kjd/idna ) >>>>>> compared to Django's 66k ( https://github.com/django/django ). >>>>>> >>>>>> I imagine most idna installs are from popular packages depending on >>>>>> it: at least requests, and twisted do. Most people probably don't care >>>>>> about complete domain validation (until they do). To me, it seems like a >>>>>> reasonable dependency for Django to adopt, given so much of the ecosystem >>>>>> uses it. >>>>>> >>>>>> But I don't think there's much evidence of demand. I found one other >>>>>> ticket mentioning idna, and this was with an extended regex, not the >>>>>> package ( https://code.djangoproject.com/ticket/18119 , seems stale, >>>>>> maybe closeable? ). >>>>>> >>>>>> I'd suggest at this point: implement idna validators in a third party >>>>>> package, do some advocacy for why projects would need it, and show some >>>>>> adoption. This would make the case for Django itself stronger. >>>>>> >>>>>> On Thu, Sep 1, 2022 at 4:18 PM 'Julien Bernard' via Django developers >>>>>> (Contributions to Django itself) <django-d...@googlegroups.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> I'm starting this discussion following ticket #33968 >>>>>>> <https://code.djangoproject.com/ticket/33968> recommendation in >>>>>>> comment. >>>>>>> Currently the punycode >>>>>>> <https://github.com/django/django/blob/e64919ae54933ef4840f0e27e51d9fcfd55ecf4b/django/utils/encoding.py#L212> >>>>>>> method is converting domain names from Unicode to ASCII using the >>>>>>> deprecated IDNA 2003 standard. >>>>>>> As suggested in the ticket, the method should use the idna >>>>>>> <https://pypi.org/project/idna/> package that is fully compliant >>>>>>> with the latest IDNA 2008 standard. >>>>>>> Ticket comment points that adding a new dependency is problematic, >>>>>>> however, in most case, someone using Django will end up with this >>>>>>> dependency anyway as it is widely used. Django GeoIP2 contrib uses it >>>>>>> for >>>>>>> instance. >>>>>>> Best regards, >>>>>>> Julien >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Django developers (Contributions to Django itself)" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to django-develop...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/django-developers/d4331b21-d9e8-4647-aa1f-357b00fb0125n%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/django-developers/d4331b21-d9e8-4647-aa1f-357b00fb0125n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Django developers (Contributions to Django itself)" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to django-develop...@googlegroups.com. >>>>> >>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/django-developers/90f58142-7808-4136-a06d-8ba1352857c7n%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/django-developers/90f58142-7808-4136-a06d-8ba1352857c7n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers (Contributions to Django itself)" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to django-develop...@googlegroups.com. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/django-developers/81d94d7b-cd25-477c-ad11-41688b5c7aben%40googlegroups.com >>> <https://groups.google.com/d/msgid/django-developers/81d94d7b-cd25-477c-ad11-41688b5c7aben%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/af30a602-1a8b-40ee-9133-556f375e3ae6n%40googlegroups.com > <https://groups.google.com/d/msgid/django-developers/af30a602-1a8b-40ee-9133-556f375e3ae6n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAJwKpySRF4jT_Aa_iC6urB9mNO6KAf%3DizJyPj_vmiJXxZ8hkeg%40mail.gmail.com.
