Bug#1043586: dnstwist: Failing autopkgtest with python3-selenium 4.11.2+dfsg-1

2023-08-19 Thread Andres Salomon

Control: affects -1 chromium


On Sun, 13 Aug 2023 12:31:13 +0200 Carsten Schoenert 
 wrote:

> Source: dnstwist
> Version: 0~20230509-1
> Severity: important
>
> Dear Maintainer,
>
> the autopkgtest of dnstwist is failing with the most recent version 
of

> python3-selenium.
>
> This is due a internal change in Selenium upstream since version 
4.11.0.
> Upstream is using a method/component called Selenium Manager since 
then
> and as we can't ship this due it's binary form calling 
webdriver.Chrome()
> needs to be extended about the information which driver needs to be 
used.

>

This is also going to cause chromium to fail to migrate to testing. (I 
say "going to" because chromium has other issues right now, but once 
those are fixed..)




Bug#1043586: dnstwist: Failing autopkgtest with python3-selenium 4.11.2+dfsg-1

2023-08-13 Thread Carsten Schoenert
Source: dnstwist
Version: 0~20230509-1
Severity: important

Dear Maintainer,

the autopkgtest of dnstwist is failing with the most recent version of
python3-selenium.

This is due a internal change in Selenium upstream since version 4.11.0.
Upstream is using a method/component called Selenium Manager since then
and as we can't ship this due it's binary form calling webdriver.Chrome()
needs to be extended about the information which driver needs to be used.

Please see attached patch that will fix the autopkgtest so it can bee
succeeding as before. Feel free to modify the patch to your needs or
requirements.
Note: The attached patch is modifing the source of dnstwist, you
probably need to check my adjustments that makes the autopkgtest working
again as changing the source has a deep impact on the usage of this package.

Regards
Carsten

-- System Information:
Debian Release: trixie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 6.3.0-1-amd64 (SMP w/20 CPU threads; PREEMPT)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
>From f4ebc3b8f71ef516efd5f56144b5fe8fb1ef7292 Mon Sep 17 00:00:00 2001
From: Carsten Schoenert 
Date: Sun, 13 Aug 2023 12:21:37 +0200
Subject: [PATCH] Add Set-driver-binary-manually-to-chromedriver.patch

This modification is required to make dnstwist function together with
python3-selenium 4.11.2 onwards due a internal change in the upstream
Python library of Selenium.
---
 ...iver-binary-manually-to-chromedriver.patch | 42 +++
 debian/patches/series |  1 +
 2 files changed, 43 insertions(+)
 create mode 100644 
debian/patches/Set-driver-binary-manually-to-chromedriver.patch

diff --git a/debian/patches/Set-driver-binary-manually-to-chromedriver.patch 
b/debian/patches/Set-driver-binary-manually-to-chromedriver.patch
new file mode 100644
index 000..343562c
--- /dev/null
+++ b/debian/patches/Set-driver-binary-manually-to-chromedriver.patch
@@ -0,0 +1,42 @@
+From: Carsten Schoenert 
+Date: Sun, 13 Aug 2023 11:48:22 +0200
+Subject: Set driver binary manually to chromedriver
+
+The Selenium upstream project is using a method/component called
+Selenium Manager since version 4.11.0+.
+This specific part is looking out which driver can be used, as this is
+shipped currently in a binary form only it's not included in
+python3-selenium. The only usable driver in Debian is chromedriver so we
+can adjust the usage of the Selenium stuff to use the binary
+/usr/bin/chromedriver.
+
+Forwared: Not-Needed
+---
+ dnstwist.py | 9 -
+ 1 file changed, 8 insertions(+), 1 deletion(-)
+
+diff --git a/dnstwist.py b/dnstwist.py
+index 702e0ad..03b665e 100755
+--- a/dnstwist.py
 b/dnstwist.py
+@@ -367,12 +367,19 @@ class HeadlessBrowser():
+   )
+ 
+   def __init__(self, useragent=None):
++  # Upstream relies on the Selenium Manager to find the correct
++  # webdriver which isn't provided by package python3-selenium.
++  # Instead dropping the automatic detection and set the driver 
hard
++  # to chromedriver and give that option to Chrome() 
initialization.
++  from selenium.webdriver.chrome.service import Service as 
ChromeService
++  service = ChromeService(executable_path="/usr/bin/chromedriver")
++
+   chrome_options = webdriver.ChromeOptions()
+   for opt in self.WEBDRIVER_ARGUMENTS:
+   chrome_options.add_argument(opt)
+   chrome_options.add_experimental_option('excludeSwitches', 
['enable-automation'])
+   
chrome_options.add_experimental_option('useAutomationExtension', False)
+-  self.driver = webdriver.Chrome(options=chrome_options)
++  self.driver = webdriver.Chrome(options=chrome_options, 
service=service)
+   self.driver.set_page_load_timeout(self.WEBDRIVER_TIMEOUT)
+   self.driver.execute_cdp_cmd('Network.setUserAgentOverride', 
{'userAgent':
+   useragent or self.driver.execute_script('return 
navigator.userAgent').replace('Headless', '')
diff --git a/debian/patches/series b/debian/patches/series
index e7c402f..4ba2989 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -1,2 +1,3 @@
 patch-out-embedded-jquery-cdn.patch
 issue194.patch
+Set-driver-binary-manually-to-chromedriver.patch
-- 
2.40.1