jenkins-bot has submitted this change. (
https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1334890?usp=email )
Change subject: Use a separate maxlag threshold for read requests
......................................................................
Use a separate maxlag threshold for read requests
maxlag is an imperfect indicator of overall server load and does not
necessarily reflect the load caused by individual read operations.
Nevertheless, it remains useful as a general safeguard against
overloading the server.
Since March 2026, recurring maxlag-related issues have made this
increasingly problematic, with maxlag rarely dropping below 5 seconds
in recent months. Waiting for the regular maxlag threshold can
unnecessarily delay lightweight read requests during such periods.
Use a new read_maxlag threshold for read requests to reduce unnecessary
delays while retaining write_maxlag (renamed from maxlag) as a safeguard
for sustained server load.
Keep the old global option -maxlag to change write_maxlag.
Show a warning if maxlag is still used within user-config.py
and set the new write_maxlag parameter accordingly.
Update tests settings.
Bug: T421642
Change-Id: I769f487581eac8f1de79066efc4abdb04b00939d
---
M .github/workflows/doctest.yml
M pywikibot/bot.py
M pywikibot/config.py
M pywikibot/data/api/_requests.py
M tox.ini
5 files changed, 39 insertions(+), 18 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/.github/workflows/doctest.yml b/.github/workflows/doctest.yml
index ecee897..d4609fe 100644
--- a/.github/workflows/doctest.yml
+++ b/.github/workflows/doctest.yml
@@ -64,8 +64,8 @@
ssh_key: ${{ secrets.WMCS_SSH_KEY }}
- name: Generate user files
run: |
- python -Werror::UserWarning -m pwb generate_user_files
-site:wikipedia:test -user:${{ env.PYWIKIBOT_USERNAME }} -v -debug;
- echo "maxlag = 20" >> user-config.py
+ python -Werror::UserWarning -m pwb generate_user_files \
+ -site:wikipedia:test -user:${{ env.PYWIKIBOT_USERNAME }} -v -debug;
- name: doctest with pytest
id: ci_test
continue-on-error: true
diff --git a/pywikibot/bot.py b/pywikibot/bot.py
index 5064c24..3ab9aec 100644
--- a/pywikibot/bot.py
+++ b/pywikibot/bot.py
@@ -233,9 +233,11 @@
-nolog Disable the log file (if it is enabled by default).
Also disable command.log.
--maxlag Sets a new maxlag parameter to a number of seconds.
- Defer bot edits during periods of database server lag.
- Default is set by config.py
+-maxlag Sets write_maxlag to a number of seconds and increases
+ read_maxlag to at least this value. If read_maxlag is
+ set to None, it remains unchanged. Defer bot requests
+ during periods of database server lag. Defaults are
+ set by config.py.
-putthrottle:n Set the minimum time (in seconds) the bot will wait
-pt:n between saving pages.
@@ -855,9 +857,14 @@
elif option == '-daemonize':
redirect_std = value or None
daemonize.daemonize(redirect_std=redirect_std)
+ elif option == '-maxlag':
+ maxlag = int(value)
+ if config.read_maxlag is not None:
+ config.read_maxlag = max(config.read_maxlag, maxlag)
+ config.write_maxlag = maxlag
else:
# the argument depends on numerical config settings
- # e.g. -maxlag and -step:
+ # e.g. -read_maxlag, -write_maxlag or -step:
try:
_arg = option[1:]
# explicitly check for int (so bool doesn't match)
diff --git a/pywikibot/config.py b/pywikibot/config.py
index eafa564..7e47bfa 100644
--- a/pywikibot/config.py
+++ b/pywikibot/config.py
@@ -38,6 +38,9 @@
default. Editor detection functions were moved to :mod:`editor`.
.. version-added:: 11.4
The 7-zip executable variable *cmd_7zip* was added.
+.. version-added:: 11.8
+ The *read_maxlag* config variable was added; The *maxlag* config
+ variable was renamed to *write_maxlag*.
"""
from __future__ import annotations
@@ -103,10 +106,11 @@
# no longer used. The values of this dict is the Pywikibot version of
# the deprecation but unused.
_deprecated_variables = {
- 'absolute_import': '10.0.0 ',
- 'division': '10.0.0',
- 'unicode_literals': '10.0.0',
- 'textfile_encoding': '11.5.0',
+ 'absolute_import': ('10.0.0', None),
+ 'division': ('10.0.0', None),
+ 'unicode_literals': ('10.0.0', None),
+ 'textfile_encoding': ('11.5.0', None),
+ 'maxlag': ('11.8.0', 'write_maxlag'),
}
# ############# ACCOUNT SETTINGS ##############
@@ -706,7 +710,8 @@
# time is increased accordingly. The default setting is 0.1 seconds per
# https://wikitech.wikimedia.org/wiki/Robot_policy.
#
-# 'maxlag' is used to control the rate of server access (see below).
+# 'read_maxlag' and 'write_maxlag' are used to control the rate of server
+# access (see below).
# Set minthrottle to non-zero to use a throttle on read access.
minthrottle = 0.1
maxthrottle = 60
@@ -721,13 +726,14 @@
# Defer bot edits during periods of database server lag. For details, see
# https://www.mediawiki.org/wiki/Manual:Maxlag_parameter
-# You can set this variable to a number of seconds, or to None (or 0) to
+# You can set these variables to a number of seconds, or to None (or 0) to
# disable this behavior. Higher values are more aggressive in seeking
# access to the wiki.
# Non-Wikimedia wikis may or may not support this feature; for families
# that do not use it, it is recommended to set minthrottle (above) to
# at least 1 second.
-maxlag = 5
+read_maxlag = 30
+write_maxlag = 5
# Maximum of pages which can be retrieved at one time from wiki server.
# -1 indicates limit by api restriction
@@ -1002,7 +1008,7 @@
'"{name}" present in your '
f'{user_config_file} is deprecated since'
' {since} and no longer a supported configuration variable and should be'
- ' removed. Please inform the maintainers if you depend on it.'
+ ' removed{instead}. Please inform the maintainers if you depend on it.'
)
@@ -1012,7 +1018,7 @@
skipped: frozenset[str],
) -> None:
"""Check the types compared to the default values."""
- for name, value in user_config.items():
+ for name, value in list(user_config.items()):
if name in default_values:
try:
if name == 'socket_timeout':
@@ -1026,12 +1032,19 @@
user_config[name] = value
elif not name.startswith('_') and name not in skipped:
if name in _deprecated_variables:
+ since, instead = _deprecated_variables[name]
msg = DEPRECATED_VARIABLE.format(
name=name,
- since=_deprecated_variables[name]
+ since=since,
+ instead=f'; use {instead} instead' if instead else '',
)
warn('\n' + fill(msg),
_ConfigurationDeprecationWarning, stacklevel=2)
+
+ if instead is not None:
+ user_config[instead] = user_config[name]
+ del user_config[name]
+
else:
warn('\n' + fill(f'Configuration variable "{name}" is defined '
f'in your {user_config_file} but unknown. It'
diff --git a/pywikibot/data/api/_requests.py b/pywikibot/data/api/_requests.py
index 168be82..254e8c2 100644
--- a/pywikibot/data/api/_requests.py
+++ b/pywikibot/data/api/_requests.py
@@ -423,6 +423,7 @@
.. version-changed:: 11.8
Disable maxlag for meta queries and paraminfo action.
+ Use a new ``read_maxlag`` parameter for read requests.
"""
if hasattr(self, '__defaulted'):
return
@@ -432,7 +433,7 @@
raise ValueError('The mime and params shall not share the '
'same keys.')
- maxlag = config.maxlag
+ maxlag = config.write_maxlag if self.write else config.read_maxlag
if self.action == 'query':
meta = self._params.get('meta', [])
diff --git a/tox.ini b/tox.ini
index a57aae8..23af122 100644
--- a/tox.ini
+++ b/tox.ini
@@ -12,7 +12,7 @@
generate_user_files = -W error::UserWarning -m pwb generate_user_files
-site:wikipedia:test -v
# ignores: eventstreams.py (can't test data), gui.py (needs tkinter),
memento.py (has too many timeouts)
DOCTEST_IGNORES = --ignore-glob=*eventstreams.py --ignore-glob=*gui.py
--ignore-glob=*memento.py
-MAXLAG = -c "with open('user-config.py','a') as f: f.write('\nmaxlag = 90\n')"
+MAXLAG = -c "with open('user-config.py','a') as f: f.write('\nread_maxlag =
90\n')"
[testenv]
basepython =
--
To view, visit
https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1334890?usp=email
To unsubscribe, or for help writing mail filters, visit
https://gerrit.wikimedia.org/r/settings?usp=email
Gerrit-MessageType: merged
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I769f487581eac8f1de79066efc4abdb04b00939d
Gerrit-Change-Number: 1334890
Gerrit-PatchSet: 7
Gerrit-Owner: Xqt <[email protected]>
Gerrit-Reviewer: Mahveotm <[email protected]>
Gerrit-Reviewer: Xqt <[email protected]>
Gerrit-Reviewer: jenkins-bot
Gerrit-CC: JJMC89 <[email protected]>
_______________________________________________
Pywikibot-commits mailing list -- [email protected]
To unsubscribe send an email to [email protected]
