Greetings,
Below is the first draft of a proposal for the adoption of a new catalog
format for the client and server. Substantive feedback is requested by
COB of Thursday, July 23rd 2009:
pkg(5): image packaging system
CATALOG FORMAT AND CACHING PROPOSAL
1. Overview
The pkg(5) server and client catalogs currently provides a summary
view of the packages provided by a repository: the FMRIs of each
package, the last time the set of available packages changed, and
the total number of packages. The server uses this information
for publication checks, to fulfill client requests, for search
indexing and analysis, and to enable browser-based access to the
repository via the BUI (Browser User Interface). pkg(5) clients
use this information to determine what packages are available, to
validate user input, and to fulfill packaging operation requests.
1.1 History
As development of the Image Packaging System has progressed, both
the server and client have increasingly required access to more
packaged metadata as fixes and various improvements have been
implemented. This has resulted in increased demand on server and
client system resources when analyzing package metadata, and
increased processing times as well.
To address catalog performance issues, a client-side unified
catalog cache was implemented, and initially contained all known
package stems from the set of publishers defined within the image
configuration. The caching mechanism was then replaced, using a
Python dict structure designed for efficient lookups of package
information by stem or FMRI and providing an ordered list of
versions, that was then serialized to disk.
Recently, the caching was revised to use a custom, delta-encoded
text format that avoided object serialization as that created an
implicit dependency on object versions and definitions, as well as
significant overhead in the on-disk footprint. To improve package
metadata performance, a new cache format was created that factored
package manifests by the types of actions contained within, and
then stored each type of action in a separate file for each
manifest.
1.2 Challenges
Despite past improvements, significant performance improvements
are still needed for both the server and client when processing
and analyzing package metadata. The work done so far has also
only benefited the client, leaving server performance behind.
Specifically, the underlying catalog data, caching mecahnisms,
and catalog retrieval operations suffer from the following
issues:
- the catalog format used for the server and client is not
consistent and the server uses local time instead of UTC
- the client does not maintain a 1:1 copy of the server's catalog
and attributes making it difficult to verify its integrity and
complicates the logic needed to obtain updates
- the caching mechanisms implemented are not granular enough,
causing some operations to run slower than necessary as more
information than is needed is loaded and processed
- no efficient lookup mechanism exists for some of the metadata,
causing operations such as dependency calculation to require a
linear scan and retrieval of manifests
- the existing caching mechanisms require clients to retrieve
manifests for all known packages to be able to perform summary
listings of available packages (at least 65 MiB for a new build
of OpenSolaris) -- which is especially harmful to GUI clients
such as packagemanager(1)
- the existing caching mechanisms do not provide the information
needed to determine (ahead of time) what package manifests need
to be retrieved during packaging operations, which leaves pkg(5)
clients unable to provide sufficient feedback to the user during
plan creation such as number of bytes to be transferred, time
estimates, etc.
- the catalog operation and caching mechanisms offered by the
depot server are not extensible, and cannot accommodate new
metadata that may be needed to perform client operations
without a client and server revision
- the catalog and caching mechanisms do not account for
future localization needs
1.3 Goals
So then, the changes proposed within this document have the
following goals:
- unification of the server and client catalog format and code
- simplification of catalog update and retrieval mechanisms
- improved granularity and transparency in caching mechanisms
allowing operations to only retrieve the information they need
- reduction of resource requirements and processing time forserver
and client
- increase of available metadata before long-running package
operations to enable improved progress and user feedback
- improved extensibility of the catalog depot operation and the
caching mechanisms used by the client
- unification and implementation of caching mechanisms and code
for client and server
2. Proposed Changes
The changes needed to accomplish the goals listed in section 1.3
are grouped below by the type of change. It should be noted that
what is described in this document is dependent on an upcoming image
and repository format versioning proposal since these changes will
require a change to the structure of both images and repositories.
2.1 Catalog Format Changes
2.1.1 Current Catalog Format
To better understand the proposed changes, it may first be helpful
to understand the current catalog format and how it is composed.
Currently, the catalog could be viewd as being composed of three
files:
- attrs
The attrs file contains metadata about the catalog. The
server and client attrs file are text/plain, and currently
have the following content:
S Last-Modified: 2009-06-23T07:58:35.686485
S prefix: CRSV
S npkgs: 40802
The client adds this content:
S origin: <repository_uri>
The Last-Modified value is an ISO-8601 date and time in server
local time (not UTC).
- catalog
The server catalog file currently contains entries of this
format:
<type> <fmri><newline><newline>
Where type can be 'V' (version), 'C' (critical; not used), or
'R' (renamed).
As a special exception, the format of 'R' entries is:
R <src_stem> <src_version> <dest_stem> <dest_version><newline>
If a destination package is not provided for 'R', then 'NULL'
is used for the destination values.
Examples:
C pkg:/[email protected],5.11-0.111:20090507T161015Z
V pkg:/[email protected],5.11-0.111:20090508T161015Z
R foo 1.0:20090508T161015Z bar 1.0:20090509T161015Z
R baz 1.0:20090508T161015Z NULL NULL
The client catalog file contains entries of this format:
<type> pkg <fmri_stem> <fmri_version><newline>
As a special exception, the format of 'R' entries is:
R <src_stem> <src_version> <dest_stem> <dest_version><newline>
If a destination package is not provided for 'R', then 'NULL'
is used for the destination values.
Example:
V pkg foo 0.5.11,5.11-0.111:20090508T161015Z
- updatelog
While not strictly a part of the catalog, the updatelog serves
as a record of changes to the catalog allowing clients to
obtain incremental updates to a catalog instead of retrieving
the entire catalog each time.
It only exists on the server, and contains entries of this
format:
<update_type><type><space><fmri><newline>
Where 'update_type' can be '+' (there were comments at one
time referring to a '-' operation, but they were removed and
the code for it was never implemented).
Where 'type' can be 'V' (version), 'C' (critical; not used),
or 'R' (renamed).
As a special exception, the format of 'R' entries is:
R <src_stem> <src_version> <dest_stem> <dest_version><newline>
2.1.2 Proposed Catalog Format
To accomplish the goals listed in section 2.1, a new catalog
format will be adopted. This format will be used by the client
to store catalog data locally, regardless of the format used by
the repository (e.g. the repository only provides older catalog
format). Catalogs are not separated by locale since manifests are
not, and all data is assumed to be encodable using UTF-8.
Catalogs using this format will be composed of the following files:
- catalog.attrs.<name>
This file will contain a python dict structure serialized in
JSON (JavaScript Object Notation) format. The metadata within
is used to describe the catalog file and its contents using the
following attributes:
created:
The value is an ISO-8601 formatted date in UTC time
indicating when the catalog was created.
last-modified:
The value is an ISO-8601 formatted date in UTC time
indicating when the catalog was last updated.
package-count:
An integer value indicating the total number of unique
FMRI stems in the catalog.
package-version-count:
An integer value indicating the total number of unique
FMRI versions in the catalog.
update-logs:
A list of dates indicating update logs available for
catalogs for clients that support incremental catalog
updates.
version:
An integer value representing the version of the
structure used within the JSON file.
Example:
{
'created': '2005-06-14T08:00:00.686485',
'last-modified': '2009-05-08T16:10:25.686485',
'package-count': 40802,
'unique-package-count': 1706,
'update-logs': ['2008100208', '2009050816'],
'version': 1,
}
- catalog.<name>
Catalog files will contain a python dict structure serialized
in JSON (JavaScript Object Notation) format. Version entries
for each package stem are kept in ascending version order to
allow fast lookups by the client and avoid sort overhead on
load. The structure can be described as follows:
{
<publisher-prefix>: {
<FMRI package stem>: [
{
"op-time": <ISO-8601 Date and Time>
"version": <FMRI version string>,
<optional-actions>: <optional-actions-data>
}
]
}
}
Initially, the server will offer the following catalogs. Each has
its content based on a tradeoff between memory usage, loading times,
and bandwidth needs which depend on the client being used to perform
packaging operations or the operation being performed.
- catalog.base
This catalog file only contains the FMRIs of the packages that
the repository contains. Loading just this catalog is useful
when performing basic listing operations using the cli, or when
simply checking to see if a given package FMRI is valid. An
example is shown below:
{
"opensolaris.org":{
"SUNWipkg":[
{
"version":"0.5.11,5.11-0.117:20090623T135937Z"
},
{
"version":"0.5.11,5.11-0.118:20090707T220625Z"
}
],
"SUNWsolnm":[
{
"version":"0.5.11,5.11-0.117:20090623T144046Z"
},
{
"version":"0.5.11,5.11-0.118:20090707T224654Z"
}
]
}
}
- catalog.dependency
This catalog file contains the FMRIs of the packages that the
repository contains, any 'depend' actions, and any 'set' actions
for facets or variants. This information is intended to be used
during dependency calculation by install, uninstall, etc. An
example is shown below:
{
"opensolaris.org":{
"SUNWdvdrw":[
{
"version":"5.21.4.10.8,5.11-0.108:20090218T042840Z",
"actions":[
"set name=variant.zone value=global value=nonglobal",
"set name=variant.arch value=sparc value=i386",
"depend [email protected] type=require",
"depend [email protected] type=require",
"depend [email protected] type=require"
]
}
],
"SUNWthunderbirdl10n-extra":[
{
"version":"0.5.11,5.11-0.75:20071114T205327Z",
}
]
}
}
- catalog.summary
This catalog file contains the FMRIs of the packages that the
repository contains and any 'set' actions (excluding those for
facets or variants). This information is intended to be used
primarily by GUI clients such as packagemanager(1), or the BUI
(Browser UI) provided by pkg.depotd(1m) for quick, efficient
access to package metadata for listing. An example is shown
below:
{
"opensolaris.org":{
"SUNWdvdrw":[
{
"version":"5.21.4.10.8,5.11-0.108:20090218T042840Z",
"actions":[
"set name=description value=\"DVD creation utilities\"",
"set name=info.classification
value=org.opensolaris.category.2008:System/Media",
]
}
],
"SUNWthunderbirdl10n-extra":[
{
"version":"0.5.11,5.11-0.75:20071114T205327Z",
"actions":[
"set name=description value=\"Thunderbird
localization - other 15 lang\"",
"set name=srcpkgs value=SUNWthunderbirdl10n-extra"
]
}
]
}
}
To enable incremental catalog updates, an "updatelog" will also be
provided as a single, merged file that can be used to incrementally
update any of the catalogs. It is composed of the following files:
- updatelog.attrs.<logdate>
This file will contain a python dict structure serialized in
JSON (JavaScript Object Notation) format. The metadata within
is used to describe the updatelog file and its contents using
the following attributes:
created:
The value is an ISO-8601 formatted date in UTC time
indicating when the updatelog was created.
last-modified:
The value is an ISO-8601 formatted date in UTC time
indicating when the updatelog was last updated.
update-count:
An integer value indicating the total number of updates
in the updatelog.
version:
An integer value representing the version of the
structure used within the JSON file.
Example:
{
'created': '2005-06-14T08:00:00.686485',
'last-modified': '2009-05-08T16:10:25.686485',
'update-count': 2,
'version': 1,
}
- updatelog.<logdate>
This file will contain a python dict structure serialized in
JSON (JavaScript Object Notation) format. <logdate> is a UTC
date and time of the format YYYYMMDDHH.
The structure of catalog update files is nearly identical to
that of catalog files, with a few exceptions. Specifically,
each version entry contains additional elements indicating
the catalog operation, the time of the operation, and any
actions that are offered by the current catalogs for that
package if applicable to the operation.
The supported types (<op-type> in the example below) of catalog
operations are:
'add' Indicates that the corresponding FMRI and metadata
(if present) has been added to the catalog.
'remove' Indicates that the corresponding FMRI has been
removed from the catalog.
The structure can be described as follows:
{
<publisher-prefix>: {
<FMRI package stem>: [
{
"op-type": <type-of-operation>
"op-time": <ISO-8601 Date and Time>
"version": <FMRI version string>,
<optional-actions>: <optional-actions-data>
}
]
}
}
An example updatelog might consist of the following:
{
"opensolaris.org":{
"SUNWthunderbirdl10n-extra":[
{
"op-type": "remove",
"op-time": "20090218T042838Z"
"version":"0.5.11,5.11-0.75:20071114T205327Z",
}
],
"SUNWdvdrw":[
{
"op-type": "add",
"version":"5.21.4.10.8,5.11-0.111:20090524T042840Z",
"actions":[
"set name=description value=\"DVD creation utilities\"",
"set name=info.classification
value=org.opensolaris.category.2008:System/Media",
"depend [email protected] type=require",
"depend [email protected] type=require",
"depend [email protected] type=require"
]
}
]
}
}
For a detailed discussion on the choice of data format and a
performance analysis, see section 3.
2.2 Server Changes
To enable clients to retrieve the new catalog files and incremental
updates to them, the following changes will be made:
- The new catalog files will be stored in the /var/pkg/catalog
directory using the filenames described in section 2.1.2. Any
existing catalog files will be converted to the new format upon
load (using writable-root if present) and the old ones removed
(unless readonly operation is in effect).
- Operations that modify the catalog file will be changed to write
out all of the new catalogs only; the version 0 catalog will no
longer be stored or used.
- The depot server will be changed to offer an additional catalog
operation "/catalog/1/" which will be added to the output of the
"/versions/0/" operation as well. It will provide a simple GET-
based HTTP/1.1 interface for retrieving catalog and updatelog
files from the server. It will not require or use any headers
other than those normally present within a standard HTTP/1.1
transaction. However, the client api will continue to provide
the uuid, intent, and user agent headers that are provided today
for the existing "/catalog/0/" operation.
- The existing "/catalog/0/" operation will continue to be offered
by the depot server for compatibility with older clients.
- The depot server will be changed to perform a simple sanity check
when starting to verify that the packages in the catalog are
physically present in the repository and that the catalog attrs
files match the catalog files. Likewise, the updatelogs will also
be checked to verify that they are valid for the catalogs. If
any of these files are found to be not valid, a warning will be
logged and the catalog files rewritten (using writable-root if
applicable). In addition, any of the corrections made will
result in corresponding updatelog entries so that incremental
updates will not be broken for existing clients.
2.3 Client Changes
To take advantage of the new catalog format, and to improve the
performance of clients, a number of changes will need to be made
to the pkg.client.api and its supporting classes. All of the
changes proposed here should be transparent to client api
consumers.
2.3.1 Image Changes
- The image object, upon initialization, will remove the
/var/pkg/catalog directory and its contents if possible.
If this cannot be done (due to permissions), the client
will continue on. If it can be removed, a new directory
named /var/pkg/publisher be created, and publisher objects
will be told to store and retrieve their metadata from it.
- Any functions contained within the image class for the
direct storage, retrieval, updating, etc. of publisher
metadata will be moved to the pkg.client.publisher and
Catalog classes.
- A new "Catalog" object reference will be added to the
image class, which will be used to allow the api access
to catalog metadata. This object will allow callers to
ask for a specific set of catalog data for an operation
(where the allowed sets match the names of the catalogs
described in section 2.1.2). The data will then be
retrieved and stored for usage by callers as needed.
- The existing catalog caching mechanism will be removed
completely as it has been superseded by the new catalog
format.
- For performance reasons, the client api will also store
versions of each of the catalogs proposed that only
contain entries for installed FMRIs to accelerate common
client functions such as info, list, uninstall, etc.
- All api functions will be changed to retrieve the catalog
data they need instead of depending upon api consumers to
do so.
2.3.2 Catalog Retrieval and Update Changes
- If a repository only offers version 0 of the catalog format,
then the client API will retrieve it, but transform and store
the catalog in version 1 format.
- If version 1 catalog data is not available, the client api will
fallback to retrieving catalog metadata by retrieving package
manifests (as it does today). This will be transparent to
clients.
- When checking for catalog updates, the client api(s) will follow
this process for version 1 catalogs when determining if a full or
incremental update should be performed for each catalog in the
image:
* if the repository now offers a version 1 catalog, but did not do
so previously, a full catalog retrieval will be performed
* the attrs file for the catalog will be checked to see if it
has been modified by asking the repository. If a not modified
status is returned (304) then no updates are available and the
catalog will be skipped during the update process. If a new
attrs file is available, it will be retrieved.
* if the attrs file has been retrieved, its corresponding catalog
file will be checked as follows:
- If the created date in the retrieved attrs file does not
match the stored attrs file, a full catalog retrieval will be
performed as the catalog has been rebuilt. In addition, a
warning will be provided to the client that there may be
something wrong with the repository (packages may be missing,
etc.).
- If the created date matches, then the version in the new attrs
file will be compared to the original, if they do not match a
full catalog retrieval will be performed as the format of the
catalog has changed (unless the client is unable to parse that
format in which case an error will be raised).
- If the version was valid, then the last modified date in the
new catalog attrs file will be compared to the original attrs
file. If the original attrs date is newer, then a full
catalog retrieval will be performed and the user will be
warned that there may be something wrong with the repository
(packages may no longer be available, etc.). If the last
modified date in the original attrs file is the same as the
new attrs file, then no updates are available and the catalog
will be skipped. If the original attrs last modified date is
older than the new attrs last modified date, then the 'update-
logs' property will be checked to see if there are incremental
updates available.
- If the update-logs property is empty, a full catalog retrieval
will be performed with the assumption that the repository has
intentionally discarded all of its incremental update
information. If the oldest updatelog listed in the new attrs
file is newer than the last modified date of the original
attrs file, then this client has not performed an incremental
for a period long enough that the repository no longer offers
incremental updates for their version of the catalog, and a
full catalog retrieval will be performed.
- Finally, if all of the above was successful, the api will then
start the incremental update process.
- When attempting to determine what incremental catalog updates
for version 1 catalogs are available, and the repository offers
version 1 catalogs, the client api(s) will use the following
process:
* the attrs file for the updatelog will be checked to see if it
has been modified by asking the repository. If a not modified
status is returned (304) then no updates are available and the
updatelog will be skipped during the update process. However,
if this was the last updatelog listed, then a full catalog
retrieval will be performed and the user will be warned that
an incremental update could not be performed due to problems
with the repository. If the attrs file has been modified,
then it will be retrieved.
* if the attrs file has been retrieved, its corresponding
updatelog file will be checked as follows:
- If the created date in the retrieved attrs file does not
match the stored attrs file, a full catalog retrieval will
be performed as the catalog has been rebuilt and the updatelog
is no longer valid even though the logdate matches. In
addition, a warning will be provided to the client that there
may be something wrong with the repository (packages may be
missing, etc.).
- If the created date matches, then the version in the new attrs
file will be compared to the original, if they do not match,
a full updatelog retrieval will be performed as the format of
the updatelog has changed (unless the client is unable to
parse that format in which case an error will be raised).
- If the version was valid, then the last modified date in the
new updatelog attrs file will be compared to the original
attrs file. If the original attrs date is newer, then a full
catalog retrieval will be performed and the user will be
warned that there may be something wrong with the repository
(packages may no longer be available, etc.). This is done as
the date change indicates that the existing updatelog is no
longer valid and an incremental update cannot be safely
performed.
- If the original attrs last modified date is older than the
new attrs last modified date, then the corresponding
updatelog file will be retrieved. If the last modified date
in the new attrs file exactly matches the the original
updatelog attrs file, a full catalog retrieval will be
performed as the catalog has been rebuilt and the updatelog
is no longer valid even though the logdate matches. In
addition, a warning will be provided to the client that
there may be something wrong with the repository (packages
may be missing, etc.).
- Finally, if all of the above was successful, the retrieved
updatelog will be added to the update queue.
- When applying the queued catalog updates, the client api will
use this process for each updatelog:
* each corresponding catalog present in the image will be
loaded, and then any updatelog entries newer than the last
modified date of the catalog (based on op-time) will be
applied to the catalog as dicated by op-type
* if at any point, an updatelog entry cannot be applied as
directed, then a full catalog retrieval will be forced, and
the user will be warned that something may be wrong with the
repository (missing packages, etc.)
* if the updatelog is the last in the queue for a given set of
catalogs, then all previous ones will be removed as they are
no longer needed
3. Appendix
3.1 Overview
During the development of this proposal, a number of different
approaches to the storage and retrieval of catalog data were
considered. Specifically, the following formats were considered
and/or evaluated:
- manifest
A pure "manifest-style" format similar to the existing package
manifest.
- JSON
The portable JavaScript Object Notation-based format.
Size and performance characteristics for each of these formats can
be found in section 3.3.
3.2 Evaluations
3.2.1 manifest-style format evaluation
Initially, the "manifest-style" format seemed promising from a
performance and disk footprint standpoint when compared to using
JSON. A few variations of this format were attempted, and examples
of this can be seen below:
- variation 1
pkg://opensolaris.org/[email protected],5.11-0.86:20080422T230436Z
[email protected],[email protected]
- variation 2
pkg://opensolaris.org/[email protected],5.11-0.86:20080422T230436Z
depend fmri=pkg:/[email protected] type=require
depend fmri=pkg:/[email protected] type=require
- variation 3
After realising that variant and facet information was needed,
and that additional attributes might need to be accounted for in
the future, variation 3 was chosen for evaluation.
pkg://opensolaris.org/[email protected],5.11-0.106:20090131T184044Z
set name=variant.zone value=global value=nonglobal
set name=variant.arch value=sparc value=i386
depend [email protected] type=require
depend [email protected] type=require
3.2.2 JSON format evaluation
When first evaluating JSON, results on x86-based systems were very
comparable or significantly better than the manifest-based format
from both a file size and performance perspective. The following
structural variations were evaluated:
- variation 1
Variation one attempted to combine the catalog and attrs files,
but this approach was abandoned for simplicity and performance
reasons in later variations.
{
"attributes": {
"id": "556599b2-aae8-4e67-94b3-c58a07dbd91b",
"last-modified: "2009-05-08T16:10:25.686485",
"locale": "C",
"package-count: 40802,
"version: 1,
},
"packages": {
"SUNWipkg": {
"publisher": "opensolaris.org",
"versions": [
"0.5.11,5.11-0.111:20090331T083235Z",
"0.5.11,5.11-0.111:20090418T191601Z",
"0.5.11,5.11-0.111:20090508T161025Z",
],
},
},
}
- variation 2
{
"packages":{
"SUNWlang-cs-extra":{
"publisher":"opensolaris.org",
"versions":[
[
"0.5.11,5.11-0.86:20080422T230436Z",
{
"depend":{
"require":[
{
"fmri":"foo"
},
{
"fmri":"bar"
}
],
"optional":[
{
"fmri":"baz"
},
{
"fmri":"quux"
}
]
}
}
],
]
}
}
}
- variation 3
This variation was attempted due to extreme performance issues
that were seen on some lower-memory bandwidth SPARC systems
when writing JSON files. It was discovered that the simplejson
library uses a recursive call structure for iterative encoding
of python data structures and this does not perform well on many
SPARC systems.
By changing the structure to a list of lists, a decrease om
write times of 20-30 seconds was realised. However, this was
less than desirable as it meant the resulting data structure
would have to be significantly tranformed after load for use
by the package system.
[['pkg://opensolaris.org/[email protected],5.11-0.111:20090508T163711Z'],
['pkg://opensolaris.org/[email protected],5.11-0.99:20081002T152038Z',
[['require',
['[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]']]]]
]
- variation 4
This variation was struck upon after the failure of the last
with the attempt to have a data structure that was immediately
useful to the packaging system after load:
{
'opensolaris.org': {
'SUNWsongbird': [
{
'depend': {
'require': [
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]'
]
},
'version': '0.5.11,5.11-0.99:20081002T152038Z'
},
],
'SUNWstc': [
{
'version': '0.5.11,5.11-0.106:20090131T191239Z'
},
],
},
}
- variation 5
The final variation is what was chosen for final evaluation for
JSON after discussions with other team members centered around
a key point: that the catalog is essentially an action pipeline
for the client. In addition, the prior variations were either
hampered by poor serialization performance on SPARC systems or
lacked the extensibility needed for possible future attribute
additions to actions.
{
"opensolaris.org":{
"SUNWdvdrw":[
{
"version":"5.21.4.10.8,5.11-0.108:20090218T042840Z",
"actions":[
"set name=description value=\"DVD creation utilities\"",
]
}
],
}
}
3.2.3 Performance Analysis
While a performance analysis was done for each variation during the
evaluation process, only the results for the chosen variation are
shown here. Analyis was performed using a dump of the /dev repo
for builds 118 and prior consisting of 42,565 unique FMRIs.
Each format that was evaluated presented unique challenges. While
the manifest-style provided simplicity and familiarity, it became
increasingly apparent during testing that any code that was used
to parse and write it would have to be changed significantly each
time changes were made to any in-memory structures that were used
as the source. In contrast, the JSON format made it easy to re-use
the in-memory python structure as the same format to be written to
disk.
The uncompressed and gzip-compressed (provided because both Apache
and cherrypy are capable of gzip compressing requests) are shown
below for comparison. Of special note is the 'all' catalog shown
below which was created to evaluate the feasbility of having a
single catalog that provided access to all commonly needed metadata
by combining the base, dependency, and summary catalogs proposed in
section 2.1.2.
=================================================================
Size Comparison
=================================================================
Catalog Mfst. Sz. JSON Sz. Mfst. CSz. JSON CSz.
-----------------------------------------------------------------
current 2.25 MiB - 327 KiB -
base 2.86 MiB 2.00 MiB 305 KiB 246 KiB
dependency 16.44 MiB 16.45 MiB 1.4 MiB 1.4 MiB
summary 7.58 MiB 7.36 MiB 483 KiB 475 KiB
all 21.16 MiB 21.47 MiB 1.6 MiB 1.6 MiB
The time needed to read and write each format is shown below for
comparison. Several runs for each catalog were performed to verify
that the timings were consistent, and the load of each system was
checked to verify that timings were not skewed.
=================================================================
Base Catalog Timings
=================================================================
System Mfst. Wr. JSON Wr. Mfst. Rd. JSON Rd.
-----------------------------------------------------------------
mine 0.13s 0.41s 0.19s 0.05s
ipkg.sfbay 0.19s 0.58s 0.29s 0.08s
kodiak.eng 0.30s 0.99s 0.37s 0.08s
cuphead.sfbay 1.18s 3.41s 1.54s 0.33s
jurassic.eng 1.37s 3.77s 1.31s 0.46s
-----------------------------------------------------------------
Mean 0.63s 1.83s 0.74s 0.20s
=================================================================
Dependency Catalog Timings
=================================================================
System Mfst. Wr. JSON Wr. Mfst. Rd. JSON Rd.
-----------------------------------------------------------------
mine 0.42s 1.06s 1.13s 0.24s
ipkg.sfbay 0.98s 1.65s 1.70s 0.39s
kodiak.eng 0.91s 2.61s 2.22s 0.40s
cuphead.sfbay 6.05s 9.00s 8.57s 1.57s
jurassic.eng 3.87s 10.46s 6.48s 2.13s
-----------------------------------------------------------------
Mean 2.45s 4.96s 4.02s 0.95s
=================================================================
Summary Catalog Timings
=================================================================
System Mfst. Wr. JSON Wr. Mfst. Rd. JSON Rd.
-----------------------------------------------------------------
mine 0.16s 0.78s 0.58s 0.14s
ipkg.sfbay 0.33s 1.09s 0.86s 0.22s
kodiak.eng 0.35s 1.90s 1.10s 0.25s
cuphead.sfbay 2.02s 6.55s 4.41s 0.92s
jurassic.eng 1.35s 7.24s 3.34s 1.25s
-----------------------------------------------------------------
Mean 0.84s 3.51s 2.06s 0.56s
=================================================================
'all' Catalog Timings
=================================================================
System Mfst. Wr. JSON Wr. Mfst. Rd. JSON Rd.
-----------------------------------------------------------------
mine 0.51s 1.22s 1.48s 0.31s
ipkg.sfbay 1.22s 1.89s 2.30s 0.51s
kodiak.eng 1.09s 3.05s 2.93s 0.53s
cuphead.sfbay 7.35s 10.38s 11.15s 2.02s
jurassic.eng 4.57s 12.20s 8.28s 2.74s
-----------------------------------------------------------------
Mean 2.95s 5.75s 5.23s 1.22s
System Notes:
- 'mine' is an Intel Core 2 DUO E8600 with 8GiB RAM
- ipkg.sfbay is a dual Opteron 2218 with 16GiB RAM
- kodiak.eng is a SPARC64-VI box with 32GiB RAM
- cuphead.sfbay is an UltraSparc-T2 with 3GiB RAM
(likely ldom or zone)
- jurassic.eng is an UltraSPARC-III+ with 32GiB RAM
From the timings seen above, it should become apparent that JSON
serialization performance is, on average, noticeably slower when
compared to a simple manifest-style format. In particular, this
is very noticeable on lower memory-bandwidth SPARC systems.
It was discovered that the likely reason for poor serialization on
some SPARC systems is that simplejson uses a recursive function-
based iterative encoder that does not perform well on SPARC systems
(due to register windows?).
This is likely because the call stack depth for the encoder will
match that of any python structure that it encodes. During the
evaluation of possible format variations, this resulted in a
hybrid approach that combined a python dict with a simple list
of actions with the hope that further improvements could be made
to simplejson at some future date. Without this approach,
significant increases in write times were seen (20-30 seconds)
when using a pure dict-based structure.
Conversely though, JSON read performance, on average, is noticeably
faster compared a manifest-style format. In part, this is because
more work has to be performed to transform the manifest-style format
into an equivalent python data structure. Notably, there is a large
cost to sorting package versions after load (having the version data
in ascending order is extremely useful to the client).
Finally, a comparison of the heap size overhead (defined as the
difference between the size of the heap before loading a catalog
and after as measured on my x86 system) is shown for comparison
below:
=================================================================
'Heap' Overhead Comparison
=================================================================
Catalog Mfst. Sz. JSON Sz. Increase Sz. Inc. %
-----------------------------------------------------------------
base 9.16 MiB 12.45 MiB +3.29 MiB +35.92%
dependency 32.34 MiB 51.48 MiB +19.14 MiB +59.19%
summary 16.84 MiB 27.41 MiB +10.57 MiB +62.76%
all 39.87 MiB 63.88 MiB +24.01 MiB +60.22%
3.3 Conclusion
When comparing the numbers alone, it seems as though the manifest-
style format should have been chosen based solely on:
- lower memory usage (43.6% less than JSON on average)
- faster write times (1.71s on average compared to 4.01s on average
for JSON)
However, ultimately, the manifest-style format was rejected for
reasons beyond simple numbers:
- desire for a defined grammar and syntax
- required maintaining custom parsing and storage code
- not easily extensible such that if additional metadata
was needed that a protocol or file format revision might
be required
- when weighing read performance vs. write performance,
read performance was considered more important as updates
to the catalog will happen far less freqeuntly than loads
of package data (loads took 3.01s on average for manifest-
style compared to 0.73s on average for JSON or about 75.75%
longer)
Instead, the JSON format was selected for the following reasons:
- full unicode support
- well-defined grammar and structure
- supported data types almost exactly mirror python's own
native data types
- allowed easy storage of existing action data of which
catalogs are essentially a summarized view of
- a python library for the parsing and writing of JSON is
part of python 2.6+
- JSON is easily portable to other systems and myriad
tools are available to parse and write it
- it is anticipated that the performance of simplejson will
only improve over time
As a final note, the approach of using separate catalogs for each
set of data instead of a single, merged catalog was used to reduce
memory usage and the amount of data that needs to be transferred
for clients.
Cheers,
--
Shawn Walker
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
