Seb35 has submitted this change and it was merged.
Change subject: Added a mechanism to cache server name in the first step
(server recognition)
......................................................................
Added a mechanism to cache server name in the first step (server recognition)
The second step was already cached (a per-wiki LocalSettings.php is created
according to its origin files), but the first step (server recognition) was not.
This is now the case: when a wiki is requested, its core variables (identifier,
version, code path, etc.) are computed and stored in a "cache file"; next times,
the host is recognised and its data are loaded. Note the origin files can be
different from a host to another due to the dynamic hierarchical loading, so
these
origin files are themselves stored and validation is done after the loading
(if it is a cache miss, we have loaded these data for nothing but it is however
better to check cache freshness afterwards than never).
Also:
* added a JSON schema for the array stored in this new cache file
* added a parameter in MWF::readFile to avoid cache a file after reading
(needed for this new feature)
* origin files used to create a decision for the existence of a certain host
are registered in order to put them in the cache file and self-check the
cache freshness ('coreconfig' in the code)
* added a system variable $FARM storing the farm name (needed for this new
feature)
Schema change:
* renamed the property 'exec' to 'executable', more understandable
Bugs:
* fixed a bug in PHP 5.6 leading to a wrong extension loading mechanism due to
a race condition: in these case we explicitely indicate the loading mechanism
(introduced some days ago) with a parameter wgUseMyExtension, another
mechanism
could be used, because in a foreach loop where new keys are added, PHP 5 also
evaluate these new keys (at the contrary of PHP 7 probably)
* AbstractMediaWikiFarmScript::copyr was not static although it should
Change-Id: I20f56c034975ed1c39da99dfc2f4f5c8bac2a8b8
---
M docs/config/farms.json
M docs/config/farms.php
M docs/config/farms.yml
M docs/configuration.rst
M docs/farms-schema.json
A docs/versions-schema.json
M src/AbstractMediaWikiFarmScript.php
M src/MediaWikiFarm.php
M tests/phpunit/ConfigurationTest.php
M tests/phpunit/ConstructionTest.php
M tests/phpunit/MonoversionInstallationTest.php
M tests/phpunit/MultiversionInstallationTest.php
M tests/phpunit/data/config/farms.php
13 files changed, 158 insertions(+), 41 deletions(-)
Approvals:
Seb35: Verified; Looks good to me, approved
diff --git a/docs/config/farms.json b/docs/config/farms.json
index fdd911b..5ccb24c 100644
--- a/docs/config/farms.json
+++ b/docs/config/farms.json
@@ -6,7 +6,7 @@
"config": [
{
"file": "LocalSettings.php",
- "exec": true
+ "executable": true
}
]
}
diff --git a/docs/config/farms.php b/docs/config/farms.php
index 91e998a..317d726 100644
--- a/docs/config/farms.php
+++ b/docs/config/farms.php
@@ -10,7 +10,7 @@
'wikiID' => 'mywiki',
'config' => array(
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -48,7 +48,7 @@
'key' => '*',
),
array( 'file' => 'org/ExecSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -81,7 +81,7 @@
'key' => '*',
),
array( 'file' => 'com/example/ExecSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
diff --git a/docs/config/farms.yml b/docs/config/farms.yml
index f2ba1ba..82f4e2f 100644
--- a/docs/config/farms.yml
+++ b/docs/config/farms.yml
@@ -6,7 +6,7 @@
wikiID: 'mywiki'
config:
- file: 'LocalSettings.php'
- exec: true
+ executable: true
## Configuration similar to the Wikimedia farm
diff --git a/docs/configuration.rst b/docs/configuration.rst
index b8f38e1..2b34109 100644
--- a/docs/configuration.rst
+++ b/docs/configuration.rst
@@ -45,7 +45,7 @@
- file: 'PrivateSettings.yml'
key: '*'
- file: 'ExecSettings.php'
- exec: true
+ executable: true
# Internally redirect .com to .org (obviously it can also be done on the
HTTP level)
'(?<lang>[a-z]+)\.(?<family>[a-z]+)\.com':
@@ -110,7 +110,7 @@
* If there is a subkey 'key' with another value containing '*' (mandatory
character), the file content must be a dictionary where keys are MediaWiki
configuration parameters and values must be dictionaries with keys (which will
be interpreted by replacing the star by the key and by replacing other
variables) and values are the corresponding values. The corresponding priority
depends on the resulting keys. There should be also a subkey 'default'
containing only variables; any key named 'default' in the file content will be
replaced by this value.
-* If there is a subkey 'exec' with boolean value 'true', the file content is
interpreted as a raw PHP and will always be executed after all dictionaries
listed above are executed. Hence it have a super-priority, in addition of the
fact all the power of PHP can be used.
+* If there is a subkey 'executable' with boolean value 'true', the file
content is interpreted as a raw PHP and will always be executed after all
dictionaries listed above are executed. Hence it have a super-priority, in
addition of the fact all the power of PHP can be used.
In order to make to the configuration easier to read, it is adviced to only
use PHP files where it is required: definition of functions, conditionnally
define configuration parameters (if their unconditional presence is harmful).
diff --git a/docs/farms-schema.json b/docs/farms-schema.json
index 85529a7..ab0882f 100644
--- a/docs/farms-schema.json
+++ b/docs/farms-schema.json
@@ -98,7 +98,7 @@
"description": "Default key.",
"pattern": "^[a-z$-]+$"
},
- "exec":
{
+
"executable": {
"type": "boolean",
"description": "This PHP file must be directly read.",
"default": false
diff --git a/docs/versions-schema.json b/docs/versions-schema.json
new file mode 100644
index 0000000..0c382d5
--- /dev/null
+++ b/docs/versions-schema.json
@@ -0,0 +1,57 @@
+{
+ "$schema": "http://json-schema.org/draft-04/schema#";,
+ "name": "Versions cache file.",
+ "type": "object",
+ "additionalProperties": false,
+ "patternProperties": {
+ "^[a-zA-Z0-9\\._-]+$": {
+ "type": "object",
+ "description": "A particular host.",
+ "properties": {
+ "variables": {
+ "type": "object",
+ "description": "Dictionary
variable-value.",
+ "additionalProperties": false,
+ "properties": {
+ "$FARM": {
+ "type": "string",
+ "description": "Farm
name.",
+ "pattern":
"^[a-zA-Z0-9\\._-]+$"
+ },
+ "$SERVER": {
+ "type": "string",
+ "description": "Server
host (must be the same as the first key in the hierarchy).",
+ "pattern":
"^[a-zA-Z0-9\\._-]+$"
+ },
+ "$SUFFIX": {
+ "type": "string",
+ "description": "Suffix
of the wiki.",
+ "pattern":
"^[a-zA-Z0-9\\._-]+$"
+ },
+ "$WIKIID": {
+ "type": "string",
+ "description": "WikiID
of the wiki.",
+ "pattern":
"^[a-zA-Z0-9\\._-]+$"
+ },
+ "$VERSION": {
+ "type": "string",
+ "description": "Version
of the wiki.",
+ "pattern":
"^[a-zA-Z0-9\\._-]+$"
+ },
+ "$CODE": {
+ "type": "string",
+ "description":
"MediaWiki code path.",
+ "pattern":
"^[a-zA-Z0-9\\/\\._-]+$"
+ }
+ },
+ "patternProperties": {
+ "^\\$[a-z]+$": {
+ "type": "string",
+ "description": "Value
of the variable."
+ }
+ }
+ }
+ }
+ }
+ }
+}
diff --git a/src/AbstractMediaWikiFarmScript.php
b/src/AbstractMediaWikiFarmScript.php
index 328930b..ea407a4 100644
--- a/src/AbstractMediaWikiFarmScript.php
+++ b/src/AbstractMediaWikiFarmScript.php
@@ -266,7 +266,7 @@
* @param string $base Internal parameter to track the base directory.
* @return void.
*/
- function copyr( $source, $dest, $force = false, $blacklist = array(),
$whitelist = null, $base = '' ) {
+ static function copyr( $source, $dest, $force = false, $blacklist =
array(), $whitelist = null, $base = '' ) {
# Return if we are considering a blacklisted file
foreach( $blacklist as $file ) {
diff --git a/src/MediaWikiFarm.php b/src/MediaWikiFarm.php
index 2bb6be9..52de18a 100644
--- a/src/MediaWikiFarm.php
+++ b/src/MediaWikiFarm.php
@@ -55,10 +55,13 @@
protected $cacheDir = '/tmp/mw-cache';
/** @var array Configuration for this farm. */
- protected $farmConfig = array();
+ protected $farmConfig = array(
+ 'coreconfig' => array(),
+ );
/** @var string[] Variables related to the current request. */
protected $variables = array(
+ '$FARM' => '',
'$SERVER' => '',
'$SUFFIX' => '',
'$WIKIID' => '',
@@ -310,6 +313,24 @@
# For now, remove loading of one config file to improve a bit
performance
// $this->setWgConf();
+ # Cache the result
+ if( $this->cacheDir ) {
+ $variables = $this->variables;
+ $variables['$CORECONFIG'] =
$this->farmConfig['coreconfig'];
+ $variables['$CONFIG'] = array();
+ foreach( $this->farmConfig['config'] as $file ) {
+ if( is_array( $file ) && ! ( array_key_exists(
'executable', $file ) && $file['executable'] ) ) {
+ $variables['$CONFIG'][] = $file['file'];
+ }
+ }
+ $versions = $this->readFile( 'versions.php', dirname(
$this->cacheDir ), false );
+ if( !is_array( $versions ) ) {
+ $versions = array();
+ }
+ $versions[$this->variables['$SERVER']] = $variables;
+ $this->cacheFile( $versions, 'versions.php', dirname(
$this->cacheDir ) );
+ }
+
return true;
}
@@ -527,12 +548,33 @@
mkdir( $this->cacheDir );
}
+ # Shortcut loading
+ if( $this->cacheDir && ( $result = $this->readFile(
'versions.php', $this->cacheDir, false ) ) && array_key_exists( $host, $result
) ) {
+ $result = $result[$host];
+ $fresh = true;
+ $myfreshness = filemtime( $this->cacheDir .
'/versions.php' );
+ foreach( $result['$CORECONFIG'] as $coreconfig ) {
+ if( filemtime( $this->configDir . '/' .
$coreconfig ) > $myfreshness ) {
+ $fresh = false;
+ break;
+ }
+ }
+ if( $fresh ) {
+ $this->farmConfig['config'] =
$result['$CONFIG'];
+ unset( $result['$CONFIG'] );
+ unset( $result['$CORECONFIG'] );
+ $this->variables = $result;
+ $this->cacheDir .= '/' . $result['$FARM'];
+ return;
+ }
+ }
+
# Now select the right farm amoung all farms
$result = $this->selectFarm( $host, false, 5 );
# Success
if( $result['farm'] ) {
- $this->farmConfig = $result['config'];
+ $this->farmConfig = array_merge( $result['config'],
$this->farmConfig );
$this->variables = array_merge( $result['variables'],
$this->variables );
if( $this->cacheDir ) {
$this->cacheDir .= '/' . $result['farm'];
@@ -541,6 +583,7 @@
mkdir( $this->cacheDir );
}
$this->variables['$SERVER'] = $result['host'];
+ $this->variables['$FARM'] = $result['farm'];
return;
}
@@ -576,10 +619,13 @@
# Read the farms configuration
if( !$farms ) {
// @codingStandardsIgnoreStart
- if( $farms = $this->readFile( 'farms.yml',
$this->configDir ) );
- elseif( $farms = $this->readFile( 'farms.php',
$this->configDir ) );
- elseif( $farms = $this->readFile( 'farms.json',
$this->configDir ) );
- else {
+ if( $farms = $this->readFile( 'farms.yml',
$this->configDir ) ) {
+ $this->farmConfig['coreconfig'][] = 'farms.yml';
+ } elseif( $farms = $this->readFile( 'farms.php',
$this->configDir ) ) {
+ $this->farmConfig['coreconfig'][] = 'farms.php';
+ } elseif( $farms = $this->readFile( 'farms.json',
$this->configDir ) ) {
+ $this->farmConfig['coreconfig'][] =
'farms.json';
+ } else {
return array( 'host' => $host, 'farm' => false,
'config' => false, 'variables' => false, 'farms' => false, 'redirects' =>
$redirects );
}
// @codingStandardsIgnoreEnd
@@ -655,12 +701,14 @@
}
# Really check if the variable is in the listing file
- $choices = $this->readFile( $this->replaceVariables(
$variable['file'] ), $this->configDir );
+ $filename = $this->replaceVariables( $variable['file']
);
+ $choices = $this->readFile( $filename, $this->configDir
);
if( $choices === false ) {
throw new MWFConfigurationException( 'Missing
or badly formatted file \'' . $variable['file'] .
'\' defining existing values for
variable \'' . $key . '\''
);
}
+ $this->farmConfig['coreconfig'][] = $filename;
# Check if the array is a simple list of wiki
identifiers without version information…
if( array_keys( $choices ) === range( 0, count(
$choices ) - 1 ) ) {
@@ -689,10 +737,6 @@
* Set the version.
*
* Depending of the installation mode, use a cache file, search the
version in a file, or does nothing for monoversion case.
- *
- * @todo “Merge” checkHostVariables and setVersion: the key
‘deployments’, when present, should be handled as soon as possible
- * to avoid reading all ‘variables’ files; in this case the keys
in the ‘deployments’ file must be host names and will
- * directly answer whether or not the wiki exists and returns its
version.
*
* @SuppressWarnings(PHPMD.ElseExpression)
* @SuppressWarnings(PHPMD.CyclomaticComplexity)
@@ -725,6 +769,7 @@
if( is_string( $this->codeDir ) && array_key_exists(
$this->variables['$WIKIID'], $deployments ) ) {
$this->variables['$VERSION'] =
$deployments[$this->variables['$WIKIID']];
$this->variables['$CODE'] = $this->codeDir . '/' .
$this->variables['$VERSION'];
+ $this->farmConfig['coreconfig'][] =
$this->variables['$DEPLOYMENTS'];
}
# Multiversion mode – version was given in a ‘variables’ file
elseif( is_string( $this->codeDir ) && is_string(
$this->variables['$VERSION'] ) ) {
@@ -762,6 +807,7 @@
}
# Cache the version
+ $this->farmConfig['coreconfig'][] =
$this->variables['$VERSIONS'];
$this->variables['$CODE'] = $this->codeDir . '/' .
$this->variables['$VERSION'];
if( $cache ) {
$this->updateVersion(
$this->variables['$VERSION'] );
@@ -979,7 +1025,7 @@
$configFile = $this->replaceVariables( $configFile );
# Executable config files
- if( array_key_exists( 'exec', $configFile ) &&
$configFile['exec'] ) {
+ if( array_key_exists( 'executable', $configFile ) &&
$configFile['executable'] ) {
$this->configuration['execFiles'][] =
$this->configDir . '/' . $configFile['file'];
continue;
@@ -1090,7 +1136,7 @@
$configFile = $this->replaceVariables( $configFile );
# Executable config files
- if( array_key_exists( 'exec', $configFile ) &&
$configFile['exec'] ) {
+ if( array_key_exists( 'executable', $configFile ) &&
$configFile['executable'] ) {
$this->configuration['execFiles'][] =
$this->configDir . '/' . $configFile['file'];
continue;
@@ -1295,7 +1341,8 @@
}
# Search for skin and extension activation
- foreach( $settings as $setting => $value ) {
+ $settings2 = $settings; # This line is about avoiding the
behavious in PHP 5 where newly-added items are evaluated
+ foreach( $settings2 as $setting => $value ) {
if( preg_match( '/^wgUse(Extension|Skin)(.+)$/',
$setting, $matches ) && ( $value === true || $value == 'require_once' || $value
== 'composer' ) ) {
$type = strtolower( $matches[1] );
@@ -1575,9 +1622,10 @@
*
* @param string $filename Name of the requested file.
* @param string $directory Parent directory.
+ * @param bool $cache The successfully file read must be cached.
* @return array|false The interpreted array in case of success, else
false.
*/
- function readFile( $filename, $directory = '' ) {
+ function readFile( $filename, $directory = '', $cache = true ) {
# Check parameter
if( !is_string( $filename ) ) {
@@ -1590,7 +1638,7 @@
# Check the file exists
$prefixedFile = $directory ? $directory . '/' . $filename :
$filename;
- $cachedFile = $this->cacheDir !== false ? $this->cacheDir . '/'
. $filename . '.php' : false;
+ $cachedFile = $this->cacheDir !== false && $cache ?
$this->cacheDir . '/' . $filename . '.php' : false;
if( !is_file( $prefixedFile ) ) {
$format = null;
}
diff --git a/tests/phpunit/ConfigurationTest.php
b/tests/phpunit/ConfigurationTest.php
index c9cd346..bf2df9a 100644
--- a/tests/phpunit/ConfigurationTest.php
+++ b/tests/phpunit/ConfigurationTest.php
@@ -241,6 +241,7 @@
$this->assertFalse(
$config['wgUseExtensionConfirmEditQuestyCaptcha'] );
# Re-load to use config cache
+ AbstractMediaWikiFarmScript::rmdirr(
self::$wgMediaWikiFarmCacheDir . '/versions.php' );
$farm = new MediaWikiFarm(
'b.testfarm-multiversion-test-extensions.example.org',
self::$wgMediaWikiFarmConfigDir, dirname( __FILE__ ) .
'/data/mediawiki', self::$wgMediaWikiFarmCacheDir,
array( 'EntryPoint' => 'index.php' )
@@ -303,6 +304,7 @@
$this->assertEquals( 200000, $config['wgMemCachedTimeout'] );
# Re-load to use config cache
+ AbstractMediaWikiFarmScript::rmdirr(
self::$wgMediaWikiFarmCacheDir . '/versions.php' );
$farm = new MediaWikiFarm( 'a.testfarm-monoversion.example.org',
self::$wgMediaWikiFarmConfigDir, null,
self::$wgMediaWikiFarmCacheDir,
array( 'EntryPoint' => 'index.php' )
diff --git a/tests/phpunit/ConstructionTest.php
b/tests/phpunit/ConstructionTest.php
index 076b97b..8e33ea8 100644
--- a/tests/phpunit/ConstructionTest.php
+++ b/tests/phpunit/ConstructionTest.php
@@ -40,6 +40,9 @@
'suffix' => 'testfarm',
'wikiID' => '$wikitestfarm',
'versions' => 'versions.php',
+ 'coreconfig' => array(
+ 'farms.php',
+ ),
'config' => array(
array( 'file' => 'settings.php',
'key' => 'default',
@@ -52,7 +55,7 @@
'key' => '*',
),
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
);
@@ -91,6 +94,9 @@
'suffix' => 'testfarm',
'wikiID' => '$wikitestfarm',
'HTTP404' => 'phpunitHTTP404.php',
+ 'coreconfig' => array(
+ 'farms.php',
+ ),
'config' => array(
array( 'file' => 'settings.php',
'key' => 'default',
@@ -107,7 +113,7 @@
'key' => '*',
),
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
);
diff --git a/tests/phpunit/MonoversionInstallationTest.php
b/tests/phpunit/MonoversionInstallationTest.php
index f8ae216..90057f8 100644
--- a/tests/phpunit/MonoversionInstallationTest.php
+++ b/tests/phpunit/MonoversionInstallationTest.php
@@ -45,6 +45,7 @@
$this->assertEquals(
array(
'$wiki' => 'a',
+ '$FARM' => 'testfarm-monoversion',
'$SERVER' =>
'a.testfarm-monoversion.example.org',
'$SUFFIX' => '',
'$WIKIID' => '',
@@ -67,6 +68,7 @@
$this->assertEquals(
array(
'$wiki' => 'a',
+ '$FARM' => 'testfarm-monoversion',
'$SERVER' =>
'a.testfarm-monoversion.example.org',
'$SUFFIX' => 'testfarm',
'$WIKIID' => 'atestfarm',
diff --git a/tests/phpunit/MultiversionInstallationTest.php
b/tests/phpunit/MultiversionInstallationTest.php
index 6e2da15..f48ef73 100644
--- a/tests/phpunit/MultiversionInstallationTest.php
+++ b/tests/phpunit/MultiversionInstallationTest.php
@@ -65,6 +65,7 @@
$this->assertEquals(
array(
'$wiki' => 'a',
+ '$FARM' => 'testfarm-multiversion',
'$SERVER' =>
'a.testfarm-multiversion.example.org',
'$SUFFIX' => '',
'$WIKIID' => '',
@@ -85,6 +86,7 @@
$this->assertEquals(
array(
'$wiki' => 'a',
+ '$FARM' => 'testfarm-multiversion',
'$SERVER' =>
'a.testfarm-multiversion.example.org',
'$SUFFIX' => 'testfarm',
'$WIKIID' => 'atestfarm',
diff --git a/tests/phpunit/data/config/farms.php
b/tests/phpunit/data/config/farms.php
index cdd2ff0..ce1f6db 100644
--- a/tests/phpunit/data/config/farms.php
+++ b/tests/phpunit/data/config/farms.php
@@ -23,7 +23,7 @@
'key' => '*',
),
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -53,7 +53,7 @@
'key' => '*',
),
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -73,7 +73,7 @@
'key' => 'default',
),
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -93,7 +93,7 @@
'key' => 'default',
),
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -113,7 +113,7 @@
'key' => 'default',
),
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -132,7 +132,7 @@
'key' => 'default',
),
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -152,7 +152,7 @@
'key' => 'default',
),
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -171,7 +171,7 @@
'key' => 'default',
),
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -187,7 +187,7 @@
'wikiID' => '$wikitestfarm',
'config' => array(
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -261,7 +261,7 @@
'wikiID' => '$wikitestfarm',
'config' => array(
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -280,7 +280,7 @@
'key' => '*',
),
array( 'file' => 'missingfile.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -300,7 +300,7 @@
'key' => 'default',
),
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -322,7 +322,7 @@
'key' => 'default',
),
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
@@ -343,7 +343,7 @@
'key' => 'default',
),
array( 'file' => 'LocalSettings.php',
- 'exec' => true,
+ 'executable' => true,
),
),
),
--
To view, visit https://gerrit.wikimedia.org/r/322433
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I20f56c034975ed1c39da99dfc2f4f5c8bac2a8b8
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/MediaWikiFarm
Gerrit-Branch: master
Gerrit-Owner: Seb35 <seb35wikipe...@gmail.com>
Gerrit-Reviewer: Seb35 <seb35wikipe...@gmail.com>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
