Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
-+-
 Reporter:  Thomas Hooper|Owner:  David
 |  Smith
 Type:  Bug  |   Status:  closed
Component:  Utilities|  Version:  dev
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Comment (by GitHub ):

 In [changeset:"95ae37839c907d7d030f1387a003a5776593d7d7" 95ae3783]:
 {{{#!CommitTicketReference repository=""
 revision="95ae37839c907d7d030f1387a003a5776593d7d7"
 Refs #30686 -- Made django.utils.html.VOID_ELEMENTS a frozenset.
 }}}
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e3b5199a8-504ede25-2e22-4455-ad54-693612519615-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
-+-
 Reporter:  Thomas Hooper|Owner:  David
 |  Smith
 Type:  Bug  |   Status:  closed
Component:  Utilities|  Version:  dev
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Comment (by GitHub ):

 In [changeset:"3cadeea077a98367a4ed344d645df0aff243de91" 3cadeea]:
 {{{#!CommitTicketReference repository=""
 revision="3cadeea077a98367a4ed344d645df0aff243de91"
 Refs #30686 -- Removed unused regexes in django.utils.text.

 Unused since 6ee37ada3241ed263d8d1c2901b030d964cbd161.
 }}}
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018dabb4aae2-ac981527-524a-4c29-b096-1d0588adfcf5-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
-+-
 Reporter:  Thomas Hooper|Owner:  David
 |  Smith
 Type:  Bug  |   Status:  closed
Component:  Utilities|  Version:  dev
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak ):

 * status:  assigned => closed
 * resolution:   => fixed

Comment:

 In [changeset:"6ee37ada3241ed263d8d1c2901b030d964cbd161" 6ee37ad]:
 {{{#!CommitTicketReference repository=""
 revision="6ee37ada3241ed263d8d1c2901b030d964cbd161"
 Fixed #30686 -- Used Python HTMLParser in utils.text.Truncator.
 }}}
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d8300f101-d347f871-c627-4025-a4bd-53ab2c31456d-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
-+-
 Reporter:  Thomas Hooper|Owner:  David
 |  Smith
 Type:  Bug  |   Status:  assigned
Component:  Utilities|  Version:  dev
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * stage:  Accepted => Ready for checkin

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d82c00d76-a9ee910f-552f-44a9-920a-cc7fa3ef69e2-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---
Comment (by Mariusz Felisiak ):

 In [changeset:"70f39e46f86b946c273340d52109824c776ffb4c" 70f39e4]:
 {{{#!CommitTicketReference repository=""
 revision="70f39e46f86b946c273340d52109824c776ffb4c"
 Refs #30686 -- Fixed text truncation for negative or zero lengths.
 }}}
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d8268c97a-4cd28b66-3c67-4e36-91b2-4fe6d8a53dcf-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---
Comment (by Mariusz Felisiak ):

 In [changeset:"48a469395191e87d3b84ad35bae2c8b53d91ed61" 48a46939]:
 {{{#!CommitTicketReference repository=""
 revision="48a469395191e87d3b84ad35bae2c8b53d91ed61"
 Refs #30686 -- Improved test coverage of Truncator.
 }}}
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d7fd6aeb7-182fe970-d461-403e-ae7b-1575d4508242-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by David Smith):

 * needs_better_patch:  1 => 0

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d6faa9f20-74a86f1f-44e5-459c-905f-c72123e81b20-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by Sage Abdullah):

 * needs_better_patch:  0 => 1


Comment:

 Oops, sorry, should've looked more closely. The
 [https://github.com/django/django/pull/16421#discussion_r1263859265
 comment linked above] hasn't been resolved.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070189e557e641-f8368426-9032-41b8-a484-a0c5b1bdc657-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by Sage Abdullah):

 * needs_better_patch:  1 => 0


Comment:

 The PR seems to have been rebased and updated since the last update on
 this ticket. Unsetting "Patch needs improvement" if that's OK.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070189e555c3a8-ea02f6db-69dc-4ea0-8a18-dc1a3d726b6a-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by Mariusz Felisiak):

 * needs_better_patch:  0 => 1


Comment:

 Based on
 [https://github.com/django/django/pull/16421#discussion_r1263859265
 comment].

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/010701895fc0e7e7-e8f97ef2-5ad9-4223-967e-b35de9e75cab-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by Mariusz Felisiak ):

 In [changeset:"6f1b8c00d8151059cc1902a92f067bbdc1673779" 6f1b8c00]:
 {{{
 #!CommitTicketReference repository=""
 revision="6f1b8c00d8151059cc1902a92f067bbdc1673779"
 Refs #30686 -- Moved add_truncation_text() helper to a module level.
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018953b6ef1a-d68776b7-8053-4f13-9fc8-0f70e2bbefcf-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by Mariusz Felisiak ):

 In [changeset:"1d0dfc0b920916900d2770f3b5640da08af36d97" 1d0dfc0]:
 {{{
 #!CommitTicketReference repository=""
 revision="1d0dfc0b920916900d2770f3b5640da08af36d97"
 Refs #30686 -- Moved Parser.SELF_CLOSING_TAGS to
 django.utils.html.VOID_ELEMENTS
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018953b6ef35-3a918a17-7dc5-40d9-9777-237132c29bda-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by David Smith):

 * needs_better_patch:  1 => 0


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018953677d27-053d1ce8-6c90-4d64-80ab-5adf6fd151d0-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by Carlton Gibson):

 * needs_better_patch:  0 => 1


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070186c0db83fb-0bab7743-805f-40f6-9c60-e2b8c052f9c2-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by David Smith):

 * has_patch:  0 => 1


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/010701863003b064-1797b3f0-c073-4799-9ff1-c1299c67d850-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by Carlton Gibson):

 Hey David — great stuff. With that in play I won't rush to resolve.

 Thinking about Matthias' comment:

 > Last but not least: I haven't benchmarked it ever, but I have the
 suspicion that...

 Could I ask you to do a minimal `timeit`/`pyperf` comparison, so we can
 get a rough measure on the table? (It doesn't need to be perfect.)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/010701857bd1797a-39f229af-5f6a-4a23-b413-733011534fbe-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by David Smith):

 [https://github.com/django/django/pull/16421 PR ]

 I was thinking about Jon's suggestion of using the HTMLParser from the
 standard library. Since the last comments on this ticket Adam Johnson has
 also written a blog post on Truncating HTML with Python's HTMLParser which
 helped inspire my PR, see [https://adamj.eu/tech/2021/09/23/truncating-my-
 blog-posts-with-html-parser/ blog post]. (I'd cc Adam as I've mentioned
 his name, but not sure how to do that?!)

 While my PR still needs more work I thought it worth sharing as it may be
 helpful to Carlton when reviewing tomorrow.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/010701857972b15c-a26ffe05-c63f-4604-b0f6-642f90888918-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by Carlton Gibson):

 Hey Matthias — that's a very useful input. Thank you for your input.

 > So, to me strip_tags is a clear case of ​a simple implementation with
 "worse is better" characteristics.

 Let, me review what happened here tomorrow (it was a **long** while ago)
 but assuming it makes sense, `wontfix` + ''We're not accepting any
 complications to the algorithm — use ... if you need more sophistication''
 may be the neatest way all round.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018578dde9f9-29c5d867-f2e5-4192-b292-cb1f1b32dc74-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by Matthias Kestenholz):

 Hi all

 lxml is quite a heavy dependency. It works **very** well but you'll wait
 for the compilation a long time if you do not have wheels. (see
 https://pypi.org/project/lxml/#files) I think Python packaging is almost a
 non-issue these days except when it comes to transitive dependencies, and
 I wouldn't want to be in charge of specifying and updating the supported
 range of lxml versions. That being said, I encountered almost no breaking
 changes in lxml since
 
[https://github.com/feincms/feincms/commit/0ec8e834dd2e0927bb23d46ee9102716c7735add
 ~2009], I use lxml in almost all projects and I can heartily recommend it
 to anyone.

 I'm sure that the regex-based solution has some problems; I'm sorry to
 admit I haven't read the full thread but I just cannot imagine a situation
 where using `|strip_tags` without `|safe` would lead to a security issue,
 and why would you want to combine these? There's no point to mark a string
 as safe after stripping all tags. So it's only about the fact that the
 output sometimes isn't nice, something which may be fixed by converting as
 many entities to their unicode equivalents as possible and only truncating
 afterwards?

 Last but not least: I haven't benchmarked it ever, but I have the
 suspicion that running bleach or html-sanitizer during rendering may be
 wasteful in terms of CPU cycles. I only ever use the sanitizer when
 saving, never when rendering. `|strip_tags` is obviously applied when
 rendering and performs well enough in many situations.

 So, to me `strip_tags` is a clear case of
 [https://hachyderm.io/@matthiask/109545192841761218 a simple
 implementation with "worse is better" characteristics].

 I truly hope this is helpful and not just a cold shower (sorry for using
 "just" here)

 Thanks,
 Matthias

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070185788808ad-e23cf74c-66cb-4adc-8550-2a42ab0a39fd-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by Jon Dufresne):

 * cc: Jon Dufresne (removed)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070185787a5aa1-75b0508f-2d6a-4006-8d26-f8eeeb87274b-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by Carlton Gibson):

 * cc: Matthias Kestenholz (added)


Comment:

 Adding some detail after the last post, since you're looking at it David.

 There was a discussion (with various folks from html5lib, and Mozilla, and
 ...) about whether html5lib could be put on a better footing.
 I'm not sure how that panned out in the medium term. (I didn't check what
 the rhythm looks like now.)

 There was alternate talk about whether bleach (or an alternate) could
 build off `html5ever` which is the HTML parser from the Mozilla servo
 project.

 * https://github.com/servo/html5ever
 * https://github.com/SimonSapin/html5ever-python (Py03 bindings.)

 That would be pretty cool, but it was clearly a lot of work, and then 2020
 happened, so...

 The other candidate in this space in Matthias' html-sanitizer:
 https://github.com/matthiask/html-sanitizer — which is built on `lxml`.



 That's just to lay down the notes I had gathered. I'm not sure the way
 forward, but hopefully it's helpful.
 Very open to ideas though! Thanks for picking it up.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070185787661d1-c7e57374-c41b-4001-b6a7-181273e417c2-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+---
 Reporter:  Thomas Hooper  |Owner:  David Smith
 Type:  Bug|   Status:  assigned
Component:  Utilities  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by David Smith):

 * owner:  nobody => David Smith
 * status:  new => assigned


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070185785c9b91-3d584f5e-6330-4253-bbe3-d30102fabff5-00%40eu-central-1.amazonses.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+
 Reporter:  Thomas Hooper  |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Utilities  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by Carlton Gibson):

 Hi Jon,

 Thank you for the comments. I will email Will, the maintainer of Bleach,
 and ask his thoughts too. Bleach has slowed down, but that's because it's
 Stable/Mature now I would have thought.

 > ...would the stdlib HTML parser be sufficient?

 Yes. Maybe. Ideally we just thought to bring in Bleach, and with it
 html5lib since, in theory that's already working code. (Florian already
 had a Truncate prototype...)

 Anyhow... will follow-up.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.f6d3e3f7f85ac7da8e40a678c98515f8%40djangoproject.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+
 Reporter:  Thomas Hooper  |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Utilities  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by Jon Dufresne):

 > To me, at this point, html5lib essentially looks unmaintained.

 I agree with this observation. The previous main maintainer looks to have
 stopped working on the project. Responses to issues and PRs have stopped.

 > Is html5lib in trouble? If so, as a user, what are your plans, if any?
 And from that, what do you think about Django adopting it? What's the
 alternative?

 For my own projects, I'll probably continue using html5lib until its
 staleness creates an observable bug for me. I haven't hit that point yet.

 Bleach, on the other hand, looks like maintenance has slowed, but not
 stopped. I believe they have vendored html5lib to allow them to make
 changes internally. FWIW, I also still use Bleach.

 ---

 I'm not familiar with all the details of this ticket, but would the stdlib
 HTML parser be sufficient?

 https://docs.python.org/3/library/html.parser.html

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.b93513dceae4c751e6d6dee3d345a209%40djangoproject.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+
 Reporter:  Thomas Hooper  |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Utilities  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by Carlton Gibson):

 * cc: Jon Dufresne (added)


Comment:

 Paging Jon, to ask his opinion on this.

 Hey Jon, I see you've made a number of PRs to both html5lib, and bleach.

 To me, at this point, html5lib essentially looks unmaintained. I don't
 have personal capacity to give to it, as cool as it is as a project.
 Arguably we (Fellows) could allocate it _some_ time, since we spend a fair
 bit already messing around with regexes but that would be small, and we
 couldn't take it on whole, so can I ask your thoughts?

 Is html5lib in trouble? If so, as a user, what are your plans, if any? And
 from that, what do you think about Django adopting it? What's the
 alternative?

 Thanks for the thought and insight.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.aa893e8f5706cde01b3654566f92705e%40djangoproject.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+
 Reporter:  Thomas Hooper  |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Utilities  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by Carlton Gibson):

 Duplicate in #30700, with [https://github.com/django/django/pull/11660
 failing test case provided].

 I've tried contacting maintainers of HTML5lib with no success.

 I've re-opened https://github.com/django/django/pull/11633 (original regex
 based suggestion) so we can at least assess it as a possible stop-gap.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.7dfcc5d75cd5b9f5b1c825e677daef9e%40djangoproject.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+
 Reporter:  Thomas Hooper  |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Utilities  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by Carlton Gibson):

 @Claude:  Totally agree.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.fdec846b25826d62b494a8977f2b8964%40djangoproject.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+
 Reporter:  Thomas Hooper  |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Utilities  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by Claude Paroz):

 To be clear, I'm also convinced parsing is more reliable than regexes. I
 just think we have to double-think before adding a dependency, because as
 the name implies, we depend on it and therefore we must be able to trust
 its maintainers. Some guarantees about the security process and serious
 bugs fixing should be obtained. Without that, we are just outsourcing
 problems.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.44f4794a79a91260cd88a2d06d34fbee%40djangoproject.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+
 Reporter:  Thomas Hooper  |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Utilities  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by Carlton Gibson):

 > AND it's just (even with the emphasis, cough) a wrapper around the
 underlying C library, so whilst 20 months seems a long time, I'm not sure
 the release cadence is really an issue.

 OK, that last one isn't at all true. (Looking at the source it's the
 entire implementation.)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.44993918c07dcd29cdd4b56b6101920f%40djangoproject.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+
 Reporter:  Thomas Hooper  |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Utilities  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by Carlton Gibson):

 Yep Claude, absolutely.

 I think there's two difficulties we could face:

 * trying to successfully sanitize HTML with regexes.
 * (Help) Make sure html5lib-python is maintained.

 The first of these is intractable. The second not. 

 I've put out some feelers to try and find out more.

 * This is pressing for Python and pip **now**, not for us for a while yet.
 * If we look at https://github.com/html5lib/html5lib-python/issues/361 it
 seems there's some money on the table from tidelift potentially.
 * We COULD allocate some time in a pinch I think.
 * AND it's **just** a wrapper around the underlying C library, so whilst
 20 months seems a long time, I'm not sure the release cadence is really an
 issue.

 BUT, yes, absolutely. Let's hammer this out properly before we commit. 
 I will open a mailing list thread when I know more.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.473885d8f73b399665516e6e426002f5%40djangoproject.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+
 Reporter:  Thomas Hooper  |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Utilities  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by Thomas Hooper):

 Hi Carlton, that would be fun, but this is bigger than I have time for
 now. It looks like you all have it in hand.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.47ce7d2e2784e70d7d2822cfb4afc9f5%40djangoproject.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser.

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+
 Reporter:  Thomas Hooper  |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Utilities  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by Claude Paroz):

 Do we want to make both html5lib and bleach required dependencies of
 Django?
 html5lib latest release is now 20 months ago, and when I read issues like
 https://github.com/html5lib/html5lib-python/issues/419 without any
 maintainer feedback, I'm a bit worried. What about the security report
 workflow for those libs? What if a security issue is discovered in html5
 lib and the maintainers are unresponsive? Sorry to sound a bit negative,
 but I think those questions must be asked.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.d603d6b28da9782f448c72f0e8e723d7%40djangoproject.com.

Re: [Django] #30686: Improve utils.text.Truncator to use a full HTML parser. (was: Truncator.chars splits HTML entities)

[Django]

#30686: Improve utils.text.Truncator  to use a full HTML parser.
---+
 Reporter:  Thomas Hooper  |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Utilities  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by Carlton Gibson):

 * version:  2.2 => master
 * stage:  Unreviewed => Accepted


Old description:

> I'm using Truncator.chars to truncate wikis, and it sometimes truncates
> in the middle of  entities, resulting in 'some text '

New description:

 Original description:

 > I'm using Truncator.chars to truncate wikis, and it sometimes truncates
 in the middle of  entities, resulting in 'some text '

 This is a limitation of the regex based implementation (which has had
 security issues, and presents an intractable problem).

 Better to move to use a HTML parser, for Truncate, and strip_tags(), via
 html5lib and bleach.

--

Comment:

 Right, good news is this isn't a regression from
 7f65974f8219729c047fbbf8cd5cc9d80faefe77.

 * The new example case fails on v2.2.3 
 * The suggestion for the regex change is in the part not changed as part
 of 7f65974f8219729c047fbbf8cd5cc9d80faefe77. (Which is why the new case
 fails, I suppose :)

 I don't want to accept a tweaking of the regex here. Rather, we should
 move to using `html5lib` as Florian suggests.
 Possibly this would entail small changes in behaviour around edge cases,
 to be called out in release notes, but
 would be a big win overall.

 This has previously been discussed by the Security Team as the required
 way forward.
 I've updated the title/description and will Accept accordingly.

 I've attached an initial WIP patch by Florian of an `html5lib`
 implementation of the core `_truncate_html()` method.

 An implementation of `strip_tags()` using `bleach` would go something
 like:

 {{{
 bleach.clean(text, tags=[], strip=True, strip_comments=True)
 }}}



 Thomas, would taking on making changes like these be something you'd be
 willing/keen to do? If so, I'm very happy to input to assist in any way.
 :)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.365d1302d9426793bc68a55f94756a7f%40djangoproject.com.